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Abstract 

Polygons are described as almost-convex if their perimeter differs from 
the perimeter of their minimum bounding rectangle by twice their 'con- 
cavity index', m. Such polygons are called m-convex polygons and are 
characterised by having up to m indentations in the side. We use a 'divide 
and conquer' approach, factorising 2-convex polygons by extending a line 
along the base of its indents. We then use the inclusion-exclusion princi- 
ple, the Hadamard product and extensions to known methods to derive 
the generating functions for each case. 



1 Introduction 

This is the second in a series of papers that look at families of m-convex self- 
avoiding polygons (SAPs). In the first [1] we outlined the 50-year history of 
polygon enumeration on the square lattice before enumerating the m = 1 case. 
This began when Temperley [2] defined enumeration problems involving self- 
avoiding walks (SAWs) that are necessarily closed, forming SAPs. Exact results 
in this area have generally been difficult to obtain, whereas asymptotic results 
have been more numerous. For example, Hammersley [3] showed that there 
exists a certain exponential asymptotic growth in the number of SAWs, counted 
by their length, and SAPs, counted by their perimeter, which is known to be 
the same for a given lattice. Furthermore, for length n, it is believed that their 
asymptotic behaviour is described by fi^ri'~^, where ^ is the growth constant 
and 7 the critical exponent. 

Exact results have so-far required the restriction of SAPs to subclasses that 
are in some way convex. In two dimensions, convexity means that the perimeter 
is equal in length to the minimum bounding rectangle (MBR). Column-convexity 
means that any vertical cross-section may only intersect the the polygon twice, 
such that all columns are connected. Examples of convex and column-convex 
polygons can be seen in Figure 1. In 1997, Bousquet-Melou and Guttmann [4] 
(BMG) gave exact results for convex SAPs of three dimensions and a method 
for their enumeration in an arbitrary dimension [4], which was based on the 
principle of inclusion-exclusion. This became central to the methods used in the 
previous paper [1], and will be one of several methods used throughout this 
paper. 
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(a) A pyramid. (b) A unimodal, or directed 

convex polygon. 



(c) A convex polygon. (d) A column-convex polygon. 

Figure 1: Some examples of convex polygons that are in some way convex. 

Enting et al. [5] described polygons as almost-convex if their perimeter dif- 
fers from the perimeter of their minimum bounding rectangle by twice their 
'concavity index', m. Such polygons are called m-convex polygons and are char- 
acterised by having up to m indentations in the side. Examples of 1-convex 
and 2-convex polygons can be found in Figure 2. Enting et al. proceeded to 
derive the asymptotic behaviour of the number of m-convex polygons according 
to their perimeter, n for m = o{^/n). The results were confirmed for the case 
m = (i.e. convex polygons) by the known perimeter generating function. Soon 
after their paper was submitted, Lin [6] derived the exact generating function 
for 1-convex polygons, using a 'divide and conquer' technique introduced to the 
problem of convex animals (the interior of a convex SAP) by Klarner and Rivest 
[7]. His result provided support for a conjecture in [5], giving the next term in 
the asymptotic expansion. The previous paper [1] then re-derived the generating 
functions for 1-convex polygons in an effort to generalise the methodology and 
extended the results to osculating^ and neighbour- avoiding^ polygons. 

So why are we interested in almost-convex polygons? Column-convex poly- 
gons are asymptotically more numerous than our almost-convex polygons. So 
if we are trying to generalise to obtain SAPs, shouldn't that be a faster av- 
enue of investigation? The answer to this question lies in the fact that we are 
interested in understanding how the generating functions change as we grad- 
ually relax the convexity restriction in all directions. For example, as deeper 
indentations are allowed in the side of the polygons, we may start to see dense 
and complex shapes that become fractal patterns in the limit as the hole depth 
grows significantly fast with respect to the perimeter. Column-convex polygons, 

^Osculating polygons are those that may touch themselves, but not cross. 
^Neighbour-avoiding polygons are those that may not occupy a neighbouring lattice vertex 
without being connected by an arc. 
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(a) A 1-convex polygon. 



(b) A 2-convex polygon. 



Figure 2: Some examples of almost-convex polygons, with the minimum bound- 
ing rectangle marked. 

in contrast, lose the convexity restriction in one direction altogether, and we do 
not learn anything about the complex behaviour of those polygons that are the 
least convex. We would therefore like to predict how the form of the generating 
functions changes as the concavity index grows. This will give us a means of 
understanding what happens in the scaling limit as the concavity index grows 
in proportion to the perimeter. 

In this paper, we start be revisiting the factorisation used by Lin [6] for 
almost-convex polygons. We then introduce the methodology used in our re- 
sults, followed by examples of its application in re-deriving 1-convex generating 
functions. We then enumerate all the separate building blocks required in the 
factorisation of 2-convex polygons, before enumerating 2-staircase, 2-unimodal 
and 2-convex polygons using the 'divide and conquer' approach. 

The Temperley method is central to the enumeration of partially convex 
polygons. The so-called 'functional- Temperley' method allowed Bousquet-Melou 
[8] to enumerate classes of column-convex polygons. It differs from the Temper- 
ley method in that it can be used to concatenate several large enumerable factors 
rather than individual columns, such that the building blocks may be character- 
istically different. A variation [9] allowed for the enumeration of certain classes 
of animals, represented as heaps of dimers. Rechnitzer [10] identified these meth- 
ods as equivalent, the superiority of one over the other lying in its ease of use 
and appropriateness to the recurrence relation underlying the problem. 

The approach that has been used repeatedly by Lin [11-13] in the con- 
struction of convex polygons is to build them up vertically, block by block. 
The functional- Temperley method can therefore be used in this case. One tool 
which can be used to 'join' polygonal blocks together is the Hadamard product 
(see [10]). This is particularly useful when a few blocks need to be joined in a 
non-recurring manner. It means that polygons with a side length enumerated 
according to a certain parameter can be joined along those sides, removing the 
arcs along the respective side perimeters and creating a larger polygon. We will 
use the Hadamard product in calculating most of the generating functions in 
this paper. 

In enumerating 1-convex polygons, Lin ran a line along the bottom of the 
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indent of a 1-convex polygon, such that it is factored into three distinct parts. 
Assuming that the indent occurs on the top, he divided all the possible polygons 
into six cases, up to vertical symmetry. In Section 2 we extend these cases to 
apply to an indent of depth m, which we call 'bimodal' m-convex polygons, as 
the path has two modes in the direction of the indent. We then calculate the 
generating function for the cases m = 1,2 and 3. 

In Section 3, we present the methodology required when adopting a 'divide 
and conquer' approach, such that we join together blocks that are often enumer- 
ated using an inclusion-exclusion approach. We begin by recalling the definition 
of the E operator, which was central to the inclusion-exclusion approach to 
enumerating 1-convex polygons. We then introduce a notation that allows us to 
apply the inclusion-exclusion approach to distinct blocks by making perimeter 
parameters independent in different blocks. In Section 3.1, we show how we can 
sometimes avoid an inclusion-exclusion argument by inserting indents at a dis- 
tinguished vertex. Next, Section 3.2, we then highlight how the block-by-block 
consruction can be extended to 'wrap' blocks that are attached to those enumer- 
ated using the inclusion-exclusion approach. Then, in Section 3.3, we present 
the Hadamard product, which is required to carry out the joining of such blocks. 
Finally, in Section 3.4, we enumerate the various staircase and unimodal blocks 
which we will need according to their side perimeters. 

Section 4 serves as an example for the use of the above techniques. We 
rederive the 1-convex polygon generating functions in a manner that is far more 
concise than the pure inclusion-exclusion approach adopted in [1]. We then 
factorise 2-convex polygons by extending a line along the base of its indents, and 
then derive the generating functions for 2-staircase, 2-unimodal and 2-convex 
polygons, in Sections 5, 6 and 7 respectively. We will see that as the examples 
get more complicated, requiring more subtle inclusion-exclusion arguments, the 
combination of Lin's factorisation, the Hadamard join and our general inclusion- 
exclusion arguments will allow for the enumeration of any arbitrary family of 
SAPs that has a fixed factorisation. The intermediary results, as well as much 
of the detail, are omitted for reasons of conciseness and clarity. These may be 
found in [14], where the presented results first appeared. 

The result of this paper is that we now have all the pieces that we need to 
enumerate m-convex polygons where each indentation has a pyramid shape, that 
is, there are no turns or hooks. The next challenge, and the most interesting 
outstanding problem, is to enumerate 3-convex polygons that have only one 
indent, but an indent that forms a hook. It will be interesting to see what 
happens to the denominator of the generating function in this case. This will 
appear in the next paper in this series. Going forward, it is not realistic to 
factorise almost-convex polygons for high concavity indices as we have done 
here, as there will be a polynomial growth in the number of cases to evaluate. 
It would be more sensible to restrict the size of each indentation first, and then 
generalise these cases. Eventually, defining operators that can add more and 
more complex indentations in the side of convex polygons and looking at the 
effect of the asymptotic growth of their number seems to be the path of least 
resistance in learning more about how convex polygons become general SAPs. 
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2 Bimodal polygons 



The generating function for pyramids, counting the widtli (by x) and the height 
(by y), also given according to the base, n, of the polygons, was given as 



n>l 



xy{l — x) 



y 



where we write 



E. 



{y} 



y 



1-y 



(1) 



(2) 



in terms of the operator E given by Bouquet-Melou and Guttmann [4]. The 
generating functions for unimodal and convex polygons were respectively given 
as 

< — , 'rni 

(3) 



and 



H{x,y) = ^Hr, = ^ 

n>l VA 

xyAc{x, y) Ax^y"^ 



C{x,y) = Y,Cn 



(4) 



where Ac = 1 — "ix — "iy -\- + + bxy — x" — y' 



A2 A3/2 ' 

^ xy{x + y + {x — y)^) and 



n>l 

A has the usual definition, 1 — 2x — 2y ^ 2xy + + y^. 

We note that solutions that can be expressed purely in terms of rational 
functions and the operator E will be rational in terms of u and v, where 



X 



u{l — v) and y = v{\ — u). 



The generating functions of the latter two classes of polygon can therefore be 
given according to their base as: 



Hn - E{y} 



- -x-y) 
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2xy u 
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and 



y\l-yf{l-x-y)\{l+y) 
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Ax'^y^u^ 
SP ■ A3/2 



(5) 



, (6) 



where SP = (1 — x — y — v A)/2 is the staircase polygon generating function. 
We denote the generating function of these functions of exact height m to be 
here X is either P, H, or C, while = X„ - YT=i ^n, 

jji is that of 

polygons of at least height m. 



These cases have a cardinality of two, due to the vertical symmetry. Referring 
to Figure 3, there are two symmetrical cases that are double counted in cases 
1 and 6, if the top factor is of height rn. We also note the symmetry of case 
8. The generating functions of bimodal m-convex polygons, which we denote 
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a,b,c,d,e > 1 



b,c,d,e > 1 



(a) Case 1. 



(b) Case 2. 



a,b,c,d > / 
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>1 



(c) Case 3. 



a b c 



o,b,c,d > 1 



(e) Case 5. 



i 1- 

b 



b.cd.e > / 



(g) Case 7. 



U_J 

a c d 
1 b.cd > 1 I 



m 



(d) Case 4. 



m 



a c 
1 a,b,c,d,e ^ 7 



(f) Case 6. 



m 



* c d 

*,r,rf > 1 



(h) Case 8. 



be d 

1 b.cd > 1 



(i) Case 9. 



Figure 3: The cases used to construct bimodal m-convex polygons following 
Lin's method. 
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as Gi,i = 1 . . . 9 for the nine classes given by Lin's factorisation, are therefore 
given by 

a,b,c,d,e>l 

G2 = ^ 2x~''~'^Pb,mHa+b+c+dHj;_^^„^ 

a>0,b,c,d,e>l 

G3 = Yl '^^~^~'^Ha+b,mPb+c+d{Hjm- Pd,m) 
a,6,c,d>l 

G4 = J2 '^^~'"'Pb,mHa+b+c+d{HX^-P2,J 

a>0,b,c,d>l 

G5 = ^ 2x ^ ^Ha+b,mHb+c+d+ePilm+l 

a,b,c,d>l,e>0 

Ge = X] X~''~'^Pb,mRa+b+c+d+e{'^Pd,m ~ Pd,m) 

a, e>0,6,c,d>l 

G7 = ^ 2x"^"''(iffe,TO - -P6,m)-P6+c+d-ff(^e,m+l 
6,c,d,e>l 

G, = ^-'-\Hb,rn-Pb,rn)Pb+cUmtrn-Ptrn)-iHd,rn-Pd,rn)) 

b, cA>l 

G9 = Y '^^~''~'^iHb,m- Pb,m)Pb+c+d+ePd,m+l- 

b,cA>l,e>0 

Implementing the sum using any of the standard computer algebra programs 
then verifies the result for 1-convex polygons. The generating function for bi- 
modal 2-convex polygons is 



(1 - - - y)A5/2 {1 - x)^{{l - x)"^ - y)^A^ 

where 

^(2) = _ 4(1 _ + 2(1 - xf{10 + 9x)y - 2(1 - xf{20 + 21 + 15x'^)y'^ 
+ 2(1 - xf{40 + 17x + 18a;2 + 21x^)y^ - (1 - a;)^(20 - 26a; - 31x'^ 
- Ux^ + 3x^)y'^ + (1 - x)(2 + x)(2 - 13a; + - 7x^ - 3x^)y^ 
+ a;(4 - 11a; - 3x^ - 5x^ - x'^)y^ + x{l + xfy'^ 

=8(1 - xf^ - 4(1 - a;)i^(16 + llx)y + 8(1 - x)"(28 + 29a; + 12a;^)y2 

- 2(1 - xf'^{22A + 233a; + 158a;^ + 51a;^)y^ + 2(1 - a;)^°(280 + 189a; 
+ 144a;^ + 95a;^ + 24a;^)/ - (1 - a;)^(448 - 57a; - 56a;^ + 36a;^ + x^)y^ 

- (1 - a;)^(-224 + 345a; + 182a;^ + 336a;^ + 292a;'' + 75a;^ + &x^)y^ 

+ (1 - xf{-M + 265a; + 30a;2 + 128a;^ - 496x'' - 267x^ + 66x^ + 2x'^)y'^ 

- (1 - x)2(-8 + 89x - 132x2 - 634a;^ - 558x'' + 593x^ - 248x^ - 64x^ 
+ 2x®)/ + x(7 - 236.x - 476x2 + 962x3 _ 3gg^4 296^5 + 248x*' - 78x^ 
+ x*)7/^ + x(13 + 188x + 99x2 _ 288x3 + 247x'' - 84x5 _^ 17x^)2/^° 

- x(ll + 58x - 28x2 + 6x3 _^ ^4^yii _ ^ ^^^^-^ ^ ^^^12 
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The generating function for bimodal 3-convex polygons was obtained and its 
form is 

xA xB 

(l-x)5((l-x)2-y)VA5/2 ^ (l-.T)9((l-.T)2-y)ll7/A3' 

Generating functions for higher concavity indices can be calculated, but we shall 
see later that there is a much more efficient way of calculating the generating 
functions, whilst still adopting the general methodology. 

For reference, we point out that the form of the 1-convex generating function 

was 

2x'^yA x'^yB 
(1 - x)A5/2 + (1 _ - xf - y)A3 ' 

We note that the dominant factor asymptotically is that of A in the denomina- 
tor, whose index is not changing. This implies that this class of polygon, as a 
proportion of the total number of m-convex polygons, is asymptotically small. 
For example, if there are m separate height one indents, for large n there is 
approximately (^) ways of choosing the location of these indents, which implies 
an rn**" derivative of the convex polygon generating function. This will increase 
the index of A by m and there will be ~ n™ times as many as the bimodal 
polygons in the limit. 



3 Methodology 

Let us recall the E operator that was originally defined in [4] and was central 
to the inclusion-exclusion approach to enumerating 1-convex polygons in the 
previous paper. 

Definition 1 {Ej). For I G {1, . . . , d} , Ej acts on a Laurent series in the vari- 
ables xi, . . . ,X(i with real coefficients. Let f{xi, . . . , Xd) = ^ cini,...,nd^i^ ' ' ' ^d''- 
Then 



Ei{f{xi,...,Xd)) = < 



Equivalently, 



f{xi,...,Xd), if/ = 0, 

0'ni,...,ni-i,2ni,ni+i,...,nd^l^ ' ' ' ■^d'*' if = 

[^{ijifi^i^ ■ ■ -^Xd))] , if |/| > 1 and i G /. 



E{i}[f{xi, Xd)] = ^ [f{xi, . . . , Xi-i, y^i, Xi+i, ...,Xd) 

+ /(.Ti, . . . -^/x'i, Xi+i, . . .,Xd)] . 

Notation (E). For our 2-dimensional case, E^i^2] [f{x,y)] is denoted E[f{x,y)]. 

We now introduce a new notation, denoted "star" (*) notation, which des- 
ignates when we are holding enumeration variables constant while using the E 
operator. This allows us to apply the inclusion-exclusion approach to distinct 
blocks by making perimeter parameters independent in different blocks. As sum- 
mation is commutative with E, this allows us to evaluate series expressions more 
simply. 
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Notation (*). Any variable that is acted on by Ej which is asterisked is as- 
sumed to be independent of the variables in /. These variables are replaced by 
their original value outside the operator. For example, 



E 



1 



(1 — X — y*)(l — X — y) 



1(1 1 

+ 



For notational convenience, we denote the staircase and unimodal polygon 
generating functions to be respectively 

SP = {l-x-y - TA) /2(= u-x = v-y) and UP = xyZ, (8) 

where Z is the generating function for pairs of directed walks, such that 

Z = l/\fK=\/{l-u-v), 

and A has the usual definition, \ — 2x — 2y — 2xy + x'^ + y'^. We define the 1-deep 
indent generating function to be 

with 1 = l{y,x) = (9) 



and, for m > 1, the m-deep indent generating function is 

= 1m{x, y) = _ ^^^^ = , with = lm{y, x). (10) 

Notation (Edge, side of a polygon). In [4], staircase polygons were defined as 
those that had two roots, one in each corner of the MBR, and factored as two 
directed walks in opposite directions. Unimodal polygons were defined similarly, 
requiring that they be rooted in one corner and that there be a factorisation for 
each direction such that there are either only positive or only negative steps in 
that direction for each factor. Finally, convex polygons were defined as those for 
whom there is a unimodal factorisation for a given direction, taking the root to 
be different vertex for each direction. Each of these factors, plus any steps along 
the edge of the MBR) we refer to as an edge. For example, the walk running 
from the bottom to the top of the MBR on the left (resp. right) side of a convex 
polygon (including the bottom and top perimeters) is called the left (resp. right) 
edge. We note that the top and bottom edges are defined similarly. 

Now, each edge can be factored into two (possibly empty) directed walks. 
We refer to these factors, plus any steps running along the MBR, as a side of 
the convex polygon. That is, a convex polygon has four directed walks running 
between the sides of the MBR, and we call them sides of the polygon. We note 
that edges are equivalent to sides for staircase polygons. Also, for unimodal 
polygons rooted in the bottom left corner, the bottom edge is the bottom-right 
side, and the left edge is the top-left side. 

Without loss of generality, we will usually assume that there is a vertical 
indent on the top-left side of the polygon, and that staircase and unimodal 
polygons are rooted to the bottom left. We can therefore classify 2-convex poly- 
gons according to the relative direction and position of their indents. They are 
either bimodal, or belong to one of four distinct cases: 
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1. in the same direction, on the same edge; 



2. in the same direction, on opposite edges; 

3. in different directions, on the same side; and 

4. in different directions, on opposite edges. 

Moreover, we note that when the indents are on edges of unimodal or convex 
polygons that have two distinct sides, then they can be broken-down further 
into either adjacent or non-adjacent sides. 

3.1 Distinguishing vertices to insert indents 

The first apphcation of Lin's factorisation enables us to insert indents by first 
distinguishing a height at which the indent is to be inserted. We are able to do 
this when the indents are joined to a staircase factor. We shall see that this is 
possible because the factor of the generating function that corresponds to the 
indent is independent of its position. 

3.1.1 Distinguishing a vertical step to insert the indent 

Let us consider bimodal m-staircase polygons. So as to have a visual guide, 
we refer to Figure 3(b) and (d), which show their form, except for their being 
rooted in the bottom-left corner. And so, measuring the width of the bottom 
and indent factors from the left edge of the top factor, their generating function 
is T^(ulx)'^v, and that of the top factor is vf'v. Combining these sums gives 



We see that the factor of is independent of the height at which the indenta- 
tion is found. 

Proposition 3.1. The generating function for bimodal m-staircase polygons 
whose single m-deep indent is vertical is 



Proof. We saw above that the generating function that enumerates the indent 
is independent of the height of the indent. This is due to the simple form of 
the staircase polygon generating function with n fixed horizontal steps, which is 
u^v. The generating function can therefore be factored as the indent generating 
function, T„i, and the generating function for staircase polygons with a distin- 
guished left-most vertical step (other than the bottom one), which is given by 




(11) 



□ 
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Figure 4: The form of 2-staircase polygons with their indent on the same side 
and in the same direction. 

3.1.2 2-unimodal polygons with both indents on the left edge 

This argument can be extended to cases where there are multiple indents on the 
one side, giving staircase factors. (See Figure 4, for example.) As each indent is 
attached to a staircase factor and is independent of the height or length of each 
join, we can enumerate them by distinguishing a vertical step and multiplying by 
X. We note that this is true, no matter the shape of the top factor. Therefore, 
this approach works for both 2-staircase and 2-unimodal polygons with their 
indents on the left. 

In the 2-staircase example, we select a vertical step on the left side that is 
not the bottom-most to designate the height at which we will place the indents. 
We then multiply by to place the two indents. Therefore, the case where 
the indents are next to each other is enumerated by I'^y^-^SP /y. We then 
take the derivative again to designate the location of the second indent. This 
gives another factorisation line, either above or below the indents, as shown 
in Figure 5(a). For a given pair of heights, there are therefore two ways that 
is occurs: with the indents either at the upper or lower height. This creates 
the combined form shown in Figure 5(b). If we take the walk between the two 
distinguished vertical steps that indicate the designated heights and translate it 
horizontally to the edges of the indents, we form two boxes. When the indents 
are at the higher of the two designated heights, the bottom walk can go inside 
these boxes. Otherwise, it cannot go into either of them. 

We want to enumerate the case where the left occurrence of the two indents 
is at the lower of the two heights and the right one is higher. This implies that 
the lower walk must be able to go into the right box, no matter the order in 
which we select the two heights, but never into the left box. We will now show 
that the generating function for this is equal to the case where one can go into 
both boxes if the upper height is selected first, and one cannot go into either 
box otherwise. 

Proposition 3.2. Let us consider a unimodal polygon with two staircase regions 
with identical sides, but possibly differing widths, identified as in Figure 5(b). 
The generating function of those polygons whose bottom perimeter intersects the 
right box, but not the left, is equal to that of the polygons whose bottom perimeter 
intersects both boxes. 
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(a) The two possible configurations, depending on whether the first height se- 
lected is above or below the second. 




(b) The combined form. In both instances, the bottom walk can 
go below the two boxes formed. Only in one can it go inside. 

Figure 5: The construction of 2-staircase polygons. Two vertical steps are se- 
lected as the heights of the indents. Both indents are placed at the first choice. 




(a) The bottom perimeter intersects the right (b) The bottom perimeter intersects 

box. the left box. 

Figure 6: The 2-unimodal polygon can be factored where it intersects the boxes 
that delimit the staircase region between the possible indent positions. 
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Proof. Let us first consider the case where the bottom perimeter intersects the 
right box. This is illustrated in Figure 6(a). If we factorise there, the bottom 
and top factors are respectively enumerated by 

—1 = 21—1 and y . 

dy dy dy y 

We then consider the case where the bottom perimeter intersects the left box. 
This is illustrated in Figure 6(b). Following Proposition 3.1 mutatis mutandis, 
the bottom and top factors are respectively enumerated by 

—I and I • y . 

dy dy y 

The first generating function is twice the second, hence the difference is equal to 
that of the latter case, where the bottom perimeter intersects the left box (and 
thus has to intersect the right one as well). The difference between the classes of 
polygons generated consists of those polygons whose bottom perimeter intersects 
the right box, but not the left. □ 

Conjecture. Considering the polygons described in the above proposition, there 
exists some perimeter preserving bijectiori^ between those polygons whose bot- 
tom perimeter intersects the right box, but not the left, and those whose bottom 
perimeter intersects both boxes. 

Corollary. The generating function for 2-staircase (resp. 2-unimodal) polygons 
with vertical indents on the left side is 

ylLj^^Q (12) 

2 5/ 5y y ' ^^'^ 
where I = (SP/y)'^ and Q = SP/y (resp. UP/y). 

Remark. In Appendix A, we extend the above result to m-unimodal polygons 
with all their indents being vertical and on the left side. 

3.2 Wrapping 

The new star notation relates to the E operator. When applied to a directed 
path generating function, E converts the perimeter generating function into the 
half-perimeter generating function. Graphically, this is equivalent to folding the 
directed path at the half-way point in each direction. Indeed, the pyramid gen- 
erating function is given Section 2 as a folded directed walk, and the unimodal 
generating function is given as a folded pyramid. 

This leads us to another new notion: that of "wrapping". To explain this, let 
us consider 1-unimodal polygons formed by joining a bottom, staircase factor to 
a top, unimodal factor and an indent. The total height of the polygon is given 
by the sum of the heights of the top and bottom factors. The total width of the 
polygon is measured by summing the width of the top factor to the width of 

^That is, the horiztonal and vertical perimeters remain constant. 
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n 



(a) The form of the top factor. (b) The form of the resulting polygon. 




(c) Flipping the top factor produces a polygon with 
the indent in the corner. 



Figure 7: The form of the polygons when the top factor is narrower than d. 

the bottom factor that lies to the left of the top factor. (In the 1-unimodal case, 
the generating function for the bottom factor is therefore Zv{u/ x)'^ .) 

Now, if the join is of length d, then there are d fixed horizontal steps in each 
of the top and bottom factors that are identified, but are then removed and do 
not form part of the polygon. If there are fewer than d horizontal steps in the 
rest of the top unimodal factor, as shown in Figure 7(a), then the contribution 
to the polygon is a pyramid of width 2n — d, with a weight of x". Importantly, 
although those fixed steps are not part of the polygon itself, they do contribute 
to the weight. This is because the width of the polygon here is given by the 
top factor. When the E operator folds the walk, using the standard inclusion- 
exclusion argument, the fixed steps are still identified with those of the bottom 
factor, which is therefore folded as well. We call the effect of one factor folding 
another along the join as "wrapping". 

Notation (Wrapping). The action of the E operator is equivalent to the folding 
of a walk or polygon. When this forces the folding of a joined polygonal factor, 
as described abve, we refer to this as wrapping. 

3.3 The Hadamard product 

The Hadamard product is an operator which 'joins' generating functions. This 
means that polygons with a side length enumerated according to a certain pa- 
rameter can be joined along those sides. For example, if we enumerate staircase 
and stack polygons according to their right perimeter, total perimeter and area, 
we can join them, making the neighbouring columns overlap. Making the trans- 
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Figure 8: An example of the action of the Hadamard product. 



formation s i— > s/yq and dividing by x, so that the overlapping column is not 
double-weighted, we form unimodal polygons, as in Figure 8. 

Definition 2 (Hadamard product / join). Let us consider two series, f{t) = 
/n*" and g{t) = Ylmdnt^- denote as Qt the Hadamard product with 
respect to t. We define 

f{t)Qtg{t)=Y.fn9nt''. (13) 

n 

The restricted Hadamard product with respect to t is defined as 

fit) Qtg{t)\t=i = Y.fn9n = ^ hmmj- (14) 

n 

For notational convenience, we will refer to this as a Hadamard join (over t), or 
simply a 'join'. 

3.3.1 Hadamard arithmetic 

It is straight-forward to show the following properties of restricted Hadamard 
products. (Alternatively, see [10] for proofs.) 

It is distributive: 

fit) Qt igit) + hit)) = fit) Qt git) + fit) Qt hit) (15) 

It follows the product rule: 

S S S 

^(/(s, t) Qt gis, t)) = i—fis, t)) Qt gis, t) + fis, t) Qt i—gis, t)) (16) 

OS ds OS 

It evaluates simply at poles: 



t^kl ( 5 



k 



(18) 

t=a 
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Another useful observation is that the join with a function of the form (1 — 
at)~'^ can be simplified by the use of partial fractions. Thus, 



k=0 "' ^""^ *=" 



(19) 



We can therefore simplify joins using partial fractions. For example, we will 
often see joins of the form 



(l-s)3(l-as) 

a 2s^ a s a ( \ 1 



+ 7^-^ ^-^-7 • (20) 



2(l-a)(l-s)3 (l-a)2(l-s)2 (l-a)3Vl-s 1 - as 

If, say, a = 1/(1— u) = ujx, we therefore have a/ (1— a) = — 1/f and 1/(1 — a) 
—xj SP, which allows us to write the join as 



vs' ^ , x\f{u/x)- f{l)) 



&s fis) = 



xf'js) f'is) 
SP 2 



s=l 



{l-sY{l-v-s) SP^ 

3.4 Enumerating the building blocks 
3.4.1 Staircase and unimodal blocks 

When enumerating 2-convex polygons using Lin's factorisation, we will need 
the generating functions for staircase and unimodal blocks according to their 
top and bottom perimeters, as well as staircases according to their base and 
right perimeter. The generating function for staircase polygons that counts the 
left and right perimeters by s and t was given by Bousquet-Melou [8]. She 
enumerated according to area, defining generalised versions of modified g-Bessel 
functions. Setting gr = 1, we obtain 

T ( \ (l-s)(l-s|/) . . xsJq{s) 

-^'^'^ = {1 - s){l - sy) + xs ^'^'^'^ = T^- 

Denoting T{s, t) = T(x, y, s, t) as the generating function according to the width 
(by x), height (by y), and right and left perimeters (by s and t respectively), we 
solve the recurrence relation given in [8] and obtain 

T{s,t) =x(^Ms)j^ + (1 - ^o(^))^) . (21) 

For notational convenience, we define T{s,t) = T{y, x, s,t). 

We similarly define the generating function for unimodal polygons according 
to their left and right perimeter and, through a slight abuse of notation, denote 
it as U{s,t) = U{x,y, s,t). From [8] one obtains 

(22) 
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Figure 9: Joining a staircase polygon to a Ferrers diagram. 



1 



2 



1 



2 



1 



(a) When the indent is not the highest factor. 



r 



2 



1 



2 



1 



(b) When the indent is the 
highest factor. 



Figure 10: The factorisation of a pyramid with a horizontal indent. 



with U{l,t) =xvt{l + SPZ)/{l-vt). 

Now, we will also need to enumerate staircases according to their base and 
right perimeter, which we do by joining a Ferrers diagram to a staircase polygon, 
as in Figure 9. Denoting the generating function s) = F{x, y, b, s), we have 



1 — bx — sy 1 — bu 



(23) 



which is in a simple form to join over b. 



3.4.2 Pyramids with indents 

In the enumeration of 2-unimodal polygons, we will need to enumerate pyramid 
factors with indents. A vertical indent we can enumerate by simply taking the 
derivative to distinguish a height, and multiplying by x^/ (1 — x)^, and so we only 
need to enumerate those with a horizontal indent. In Figure 10, we illustrate 
the two cases of such a pyramid: when the indent factor either does or does not 
touch the top of the MBR. The generating function, which we denote P'{x,y), 
is therefore 



P'{x,y) 



xy 



y 

y \l-x-y 



x{l 
(1^ 



Xj 



+ 



xy 



(24) 
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(a) Non-pyramid 
unimodal polygons. 



(b) Non-unimodal convex 
polygons that touch left first. 



(c) Non-unimodal convex 
polygons that touch both sides 
at the same height. 



Figure 11: A decomposition of the form of convex polygons that allows their 
enumeration by the base and top perimeter. 

3.4.3 Enumerating convex polygons by their base and top perimeter 

Extending the inclusion-exclusion enumeration of convex polygons of Bousquet- 
Melou and Guttmann [4] mutatis mutandis, the generating function for convex 
polygons according to the left-perimeter is 

c(s) = E [ ^^/(i-^)^(i-^)^ ( ^-y _ i] 

^' [ 2{l-x-yy \ x"^ J l-x*s-y 

- (25) 

\u J 1 — us 

Now let us consider convex polygons according to their top and bottom 
perimeters, whose generating function we denote C{s,t). We can decompose 
convex polygons into three classes: unimodal, non-unimodal that touch the sides 
of the MBR at different heights, and those that do not. The first of these classes 
contains polygon that can be classified as rooted in either of the bottom cor- 
ners. If include both of these classes, we double-count the pyramids. And so we 
enumerate unimodal polygons rooted in one corner, and non-pyramid unimodal 
polygons rooted in the other. The latter class is depicted in Figure 11(a), and 
the total generating function is 2U{s,t) — P{s,t). 

The next class are convex, but not unimodal, and include only those that 
touch the side of the MBR at different heights. We assume, without loss of gen- 
erality, that they touch the left side first, as per the form shown in Figure 11(b). 
We form these polygons by joining non-pyramid unimodal polygons (as shown 
in part (a) of the figure) with upside-down pyramids, overlapping the bottom 
rows, except for a single cell on the left. The latter conditions determines that 
the polygon touches the right side at a greater height. The generating function 
for this case is given by 

2 

— (U{rs,t) - P{rs,t)) QrE, 
rs 

Finally, the case where the polygon touches each side at the same height 
forms a diamond-like polygon, as shown in Figure 11(c). Similar to the previous 



rs y 
1 — rs 1 — r — y 
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result, the generating function is simply 



1 , xrsty\ ^ I n 




The three cases then sum to give the desired generating function: 



(1 - s(l + (1 - s)x - y))(l - t{l + (1 - t)x - y))A3/2 
— stx^y'^{l — x + y — s{l — X — y))(l — x + y — t{l — x — y)) 

stxyA{s,t) 




(1 - stx){{l - sx)2 - y)((l - txf - y)VA 



where A{s,t) = {l-s){l-t)(l-x)^{l-sxf{l-tx)^ + {l-x)^{l-sx)^{l-tx)^{-5- 
x-2x'^-t'^x{5+2x+x'^)+t{6+4x+5x'^+x^)+s'^x{-5-2x-x'^+t'^x{-4:-5x+x'^)+t{6+ 
Ax+7x'^-x^))+s{6+4:X+5x^+x^+t'^x{6+4:X+7x^-x^)+t{-7-9x-8x^-9x^+x'^)))y+ 
(10 + 3a; - 3a;2 + 5x^ + x^ + 2t^x^ (5 + + 2x^) - t^x^ (35 + 13a;2 + lOa;^ + &x'^)+ ex{m + 
12x+ 17x2 + 11x3 + 15a;'* + x5)-t(15 + 23x + 5.T2 + 8.x3 + 10x'* + 33;5) + .s''.x3(2f4.x3(3 + x + 
5x2-x3)+2(5+x2+2x3)+2t2a;(17+5x+9x2+19x3-2x'')+tV(-26-2x-27x2-12x3+ 
3x4) - 1 (15 + 16x + 7x2 + 22a;3 + 4x4)) + 533.2 (_35 _ 133.2 _ ^Qx^ - Sx^ + t^x^ (-26 - 2x - 
27x2 -12x^ + 3x4) +i3x2(100 + 5x + 79x2 + 63x3 + 13x4-4x5) +t2a;(_i23-40x- 71x2- 
105x3-50x4+5x5)+t(51+66x+23x2+71x3+38x4+7x5))+s2a;(40+12x+17x2+llx3+ 
15x4+x5+2t4a;3(17+5x+9x2+19x3-2x4)+i2x(145+92x+93x2+113x3+130x4+3x5) + 
t^x"^ (- 123 - 40x - 71x2 - 105x3 - SOx" + 5x5 ) _ ^(57 ^_ io2x + 43x2 + 79x3 _^ 78^,4 _^ 23x5 + 
2x^))-s(15 + 23x + 5x2+8x3 + 10x4 + 3x5 + i4a;3(15 + 16x+7x2 + 22x3+4x4)-t3x2(51 + 
66x+23x2 + 71x3+38x4 + 7x5)+t2a;(57+i02x+43x2+79x3 + 78x4+23x5+2x6)-t(21 + 
69x + 44x2 + 39x3 _^ 563,4 _,_ ^q^^ + 7x6)))y2 + (_ iq - 22x - x2 - 3x3 - 2^43,3 (5 + 4a; + 3x2) + 
t3x2(40+34x+29x2+9x3)-t2a;(50+62x+49x2+25x3+2x4)+t(20+52x+38x2+20x3 + 
5x4+x5)+s4x3(t4x3(-4-x+x2)-2(5+4x+3x2)+t3a;2(24+10x+9x2-3x3)+t2a;(-36- 
28x-31x2+3x3)+i(20+26x+23x2+llx3))+s3a;2(40+34x+29x2+9x3+fV(24+10x+ 
9x2-3x3)+t2a;(152 + 130x+151x2+62x3-7x'*)+t3a;2(-ii0-61x-70x2-13x3 + 6x4)- 
t(75+118x+109x2+72x3+18x4))+s2a;(_50-62x-49x2-25x3-2x4+t4a;3(_36-28x- 
31x2 + 33;3) _^ t^x'^{lb2 + 130x + 151x2 + 62x3 _ 73,4^ _ ^2 3,^200 + 234x + 261x2 + 184x3 + 
9x4) + i (90 + 190a; + 192x2 + 147x3 + 54x4 + 7x5)) + s(20 + 523. + 383.2 ^ 20x3 + 5x4 _^ 3,5 _^ 
t4a;3 (20 + 26x + 23x2 + llx^) - t^x"^ (75 + 1 18x + 109x2 + 72x3 + 18x4) + t^x{%Q + 190x + 
192x2 + 147x3 + 54x4 + 73.5) _ ^(35 + i30x + 147x2 + 121x3 + 47x4 + 23x5 + x^)))y^ + {b + 
18x + x2 - x3 + t4a;3 (5 + 4x) - ^33,2 (25 + 27x + 9x2) _^ ^23.(35 _^ 593. _^ 29x2 + 3x3) - ^(^5 _^ 
53x + 32x2 + 6x3 + 3a;4) + 343,3 (5 + 43; + ^43,3 + ^33,2 (_ ^ ^ _ 53, ^ 3,2) ^ ^23,(19 ^ I53, ^ 2x2) - 
i(15 + 17x + llx2)) + s3x2(-25-27x-9x2+f4.x3(_ii_5a; + x2)+t3a;2(65 + 45x+12x2- 
4x3)+t2a;(-103-102x-56x2 + x3)+t(65 + 98x + 80x2 + 23x3)) + .s2x(35 + 59x + 29x2 + 
3x3 ^ ^43.3(19 ^ 153, ^ 2x2) + t3a;2(-i03 - I02x - 56x2 + 3;3) _^ ex{lbb + 210x + 168x2 
+29x3) -t(85 + 180x+ 179x2 + 81x3 + 11x4)) -s(15 + 53x + 32x2 + 6x3 + 3x4 +t4a;3(i5 + 

17x+llx2)-t3a;2(65 + 98x+80x2 + 23x3)+i2a;(85 + 180x+179x2 + 81x3 + llx4)-i(35+ 
140x + 163x2 + 93x3 ^ 353,4 ^ 43.5 ) ) )y4 _^ (_ 1 _ 33. _^ 2x2 - t4a;3 + 2t3a;2 (4 + 3x) - t2a;(13 + 
21x + 4x2)+f(6 + 26x + 9x2 + 3x3) + .s4x3(-l + 2f3a;2-2f2.T(2 + x)+t(6 + 4x)) + s3x2(8 + 
6x + 2^4a;3+t33,2(_i9_8a; + a;2) + 2t2x(18 + 15x + 2x2)-f(33 + 39x+16x2)) + s23;(_i3_ 
21x - 4x2 - 2i4a;3(2 + x) + 2*3^2(18 + 15x + 2x2) _ ^23.(64 _^ 333, ^ 333.2-) ^ ^(43 ^ 903, + 
63x2 _^ 113,3)) _^ g(g _^ 26x + 9x2 ^ 33,3 _^ 2f43;3 (3 _^ 2x) - f3a;2 (33 + 39a; + 16x2) + f2.^(48 + 
90x + 63x2 _^ 113,3) -t[21 + 87x + 83x2 _^ 333,3 _^ 6x4)))?y5 + (-x - i^x^ - s4ia;3 + 2i2a;(l + 
x) - i (1 + 5x + x2) + s3a;2(-i + 2t3a;2 - 123;(5 + 2x) + 1(9 + 6x)) + s2x(2(l + x) - t3a;2(5 + 
2x) + ^2^(11 + 12x) - 1(15 + 22x + 7x2)) _ s(i + 5^ + a;2 + t4a;3 _ 3^33,2 (3 ^ 2x) + t2a;(15 + 
22x+7x2)-t(7+29x + 18x2+4x3)))y6_si(i_2(-2 + s+t)x + (-2s + s2 + (l-t)2)a;2)y7 
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4 Enumerating 1-convex polygons using Lin's factori- 
sation and the inclusion-exclusion principle 

In section we use Lin's factorisation before the inclusion-exclusion principle to 
rederive the 1-unimodal and 1-convex polygon generating functions. (The 1- 
staircase case was enumerated in the previous section by distinguishing a vertical 
step.) This contrasts greatly with the more complicated derivation in [1] that 
generated all polygons as a single case of inclusion-exclusion. These results will 
serve as an example for the general approach we adopt later in this paper. 



4.1 Enumerating 1-unimodal polygons 
4.1.1 Indent on left. 

This case was also enumerated in the previous section. Adopting the same ap- 
proach mutatis mutandis, the generating function is I ■ y-^ UP. This confirms 
the following generating function, obtained via inclusion-exclusion: 

u'^v ( „ \x'^y{\ — — y) 



E 



E 



x-y 



-u'^vE 


xy 


l-x-y_ 





vE 



xy^ 



x-y 



d>l 



X 



vxyE 



d>l 

1 



E 



xy 



x-y. 



(1 — X — v*){l — X — y) 



2 ry 
UV Z 



2xy 



(27) 



Above, the inclusion term includes polygons of width less than d, such that the 
polygon wraps, to form those shown in Figure 7(b). We can see how this allows 
us to enumerate both pyramid and unimodal top factors using the inclusion- 
exclusion method, without restricting the direction of the path as it passes 
through Lin's factorisation line. 



4.1.2 Indent in corner. 

Now, in the previous enumeration, when the top factor is a pyramid the indent 
occurs higher than all of its right edge steps. In other words, it touches the right 
edge below the factorisation line, as depicted in Figure 7(b). This means that 
we can swap the order of the indent and the top pyramid factor, as per part (c) 
of the figure, creating an indent in the top-right corner. Noting that we must 
include the case where the indent factor touches the right edge of the MBR and 
enumerating them as corner-staircase polygons from [1], we obtain 

lySPZ (^— + uZ 
\1 — X 

One can check that this is the same result as that which is obtained by adopt- 
ing the inclusion-exclusion method. To enumerate the case where the indent is 
not to the right of the bottom factor, and taking the first term of equation 27 
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(n.b. there is no possibihty of intersection), we exclude the cases of width > d 
by writing 



(1 - uf 



d>l 



1 



l-u 



_l-x-y\ \l-y 



SP' 



d>l 



■E 



1-x-y 



u 



u 



{1-x-y) 



1 — u \_\l — u*/x 1 — n*/(l — y)^ 

This is equivalent to distinguishing a vertical step at a height above the right 
edge of the polygon. 



4.1.3 Indent on bottom. 

Using our above methodology mutatis mutandis, the generating function for the 
case where the indent is on the bottom edge of the polygon is 



'^yvE{yy 



u 



{1 — u* — y){l — X — y)^ \ (1 — y)^ — X 1 — X J 



+ 



2xvSP^Z 
yA 



(28) 

Whenever indents are attached to the base of a unimodal factor, we will see 
generating functions of this form. 



4.2 Enumerating 1-convex polygons 

In the case of 1-convex polygons, not having to root the polygon in one corner, 
like we did for 1-unimodal polygons, returns certain symmetries. This means 
that we no longer have to enumerate three asymmetric cases; we only have to 
do the case where the indent is on the top-left side of the polygon. We note, 
however, that when the top factor is of height one, there is a vertical symmetry. 
And so, when the indent is at the top of the polygon, it will be double counted. 
This results in the term (1 -|- y/(l — x)) appearing in the generating function. 

Following the above methodology mutatis mutandis, the generating function 
for 1-convex polygons is 



E 



x{l-x)y* 
(1 - x)2 - y 

4a;y'u 
A~ 



(1 



E 



x^y 



1 — X* 



— y J \0- — yY — X 

(1 - y? 



+ 



x*y^ 



1 — X* J 1 — X — y 



2xyu'v 



{l-u^A 



E 



vj v(i-y)^ 

x{l — x)y^ 



x^ 



{1-x) 



y 



+2vSPZ 



2xSP 



2SP^Z 

y 



1 + 



1 

1-x 



+ 



y^ 



1 



(1-. 

+ uZ 



X 



1 — X — V* 



1 + 



l-u 



ZUP, 



where the last term comes from the possibility of intersection along the NW 
diagonal, factoring into a unimodal polygon and a 1-unimodal SAP with the 
indent in the corner. 
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(a) The indents form a convex factor. 



(b) The indents form a con- 
cave factor. 



Figure 12: The relative position of indents on adjacent sides can cause a convex 
or concave shape (in the usual non-lattice sense of convexity). 

5 Enumerating 2-staircase polygons 

In Section 3.1, we enumerated bimodal 2-staircase polygons. We will now enu- 
merate the four cases when the indents are distinct. 

5.1 Case 1: indents in the same direction, on the same edge 

In Section 3.1, we also enumerated 2-staircase polygons with both indents on 
the left edge. The case when the indents are at the same height follows from the 
1-staircase result. These give the desired result. 

5.2 Case 2: indents in the same direction, on opposite edges 

If there is one indent each on the top and bottom edges, then there are three 
different cases to enumerate: if the top one is above, level or below the bottom 
one. If they are level, the top one can be to the left (enumerated by X^vSPZ) 
or the right ((x/(l — x))^{SP/{l — u))^). The other cases are enumerated by 

wvs - wvt 

{1 - w){l - s){l - V - s) ®' ®* {l-w){l-t){l-v-t)' 

where w = u \l the top indent is above the bottom indent, and w = s or t Hit is 
below. To evaluate such expressions, we refer to the factorisation of joins using 
partial fractions in (20) and let a = 1/(1 — v) = u/x. 

5.3 Case 3: indents in different directions, on the same edge 

When the indents of a 2-staircase polygon are on the same edge, they can form 
a middle factor that is either convex or concave (in the usual Euclidean sense 
of the words). This is shown in Figure 12. The generating functions can be 
obtained by using the techniques of Section 3 mutatis mutandis. 
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(a) The symmetric case, with (b) The left indent is below the 

the left indent above. bottom indent. 




(c) The left indent is next-to the bottom indent. 



Figure 13: The relative position of indents on adjacent sides in opposite halves 
puts it in one of three cases. 

5.4 Case 4: indents in different directions, on opposite edges 

When the indents are the bottom and left, there are three possible cases in 
terms of their relative position. These are shown in Figure 13. The first case is 
symmetrical about the x = y diagonal. The other two cases have an equivalent 
class after reflection. 

The generating function for the first case can be given on inspection, from the 
corner-staircase generating function from [1]. It is XTSP^Z^ . Similarly, for both 
cases where the top indent is next-to the bottom indent, as in Figure 13(c), one 
can write down their generating functions on inspection (which, incidentally, 
are equal). Finally, when the top indent is below the bottom one, as in Fig- 
ure 13(b), the generating function is given by simply joining the staircase and 
indent factors. 
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5.5 The generating function for 2-staircase polygons 

Combining all of the above cases gives the following generating function for 
2-staircase polygons: 



where 



2xV(l - - y)^{l -x-y) 2x4y4(i _ x){l - y)A3/2 



(29) 



=(1 - xfx^ - 3(1 - xfx^iS - 6x + x2)2/ + (1 - xfx'^iA - 8x + SSx^ 

- 109x^ + 112.T^ - 47.T''^ + 3x^)]/ - (1 - x)^x^{28 - 96x + 169x^ 

- 321a;3 + 471a;'' - 395a;^ + 18Ax^' - 39a;^ + x^)y^ - (1 - a;)(-l + 8a; 

- 105a;^ + 436x^ - 890a;'' + 1198a;^ - 1292a;^ + 1120a;^ - 701a;^ + 301x'^ 

- 81a;^° + 9a;")/ + (-9 + 81a; - 394a;^ + 1144a;^ - 2088a;* + 2562a;^ 

- 2238a;'' + 1410a;^ - 588a;^ + 139a;^ - 20a;^° + 2a;" + x^^)y^ + (36 

- 318a; + 10873;^ - 2037a;3 + 2490a;* - 2238a;5 + 1530a;'5 - 738a;^ 
+ 192a;^ - 6x^ - 3x^° - x^^)y^ + (-84 + 714x - 2071x^ + 2940a;^ 

- 2412a;* + 1410a;^ - 7383;*^ + 312a;^ - 70a;^ + 6x^ - x^°)y'^ + (126 

- 1008a; + 2621.t2 - 3103^;^ + 1821a;* - 588a;^ + 192a;S - 70a;^ + 6a;^ 
+ x^)y^ - (1 - a;)-'^(126 - 546a; + 167a;2 + 18a;3 + 9a;* + x^)y^ 

+ (1 - a;)3(84 - 294a; + Sbx^ + 6x^ + x%^° + (1 - a;)3(-36 + 90a; 
+ a;2 + a;3)y" - (1 - xf{-9 + I2x + x^)y^^ - (1 - xfy^^ 
Bi'^ =(1 - x)^x^ - (1 - xf{9 - x)x^y - (1 - xfx^{-A + 8x- 33a;2 + Slx^ 
+ 13a;* + lx^)y^ - (1 - xfx^{28 - 64a; + 97a;2 - 147a;^ + 10a;* - x^ 

- Mx^ + x'^)y^ + (1 - a;)(l - 8a; + 105a;^ - 268a;^ + 326a;* - 338a;^ 

+ 1943;*^ - 55a;^ + 55a;^ - 49a;^ + 5a;^°)y* + (-9 + 73a; - 336a;^ + 658a;^ 

- 664a;* + 478a;^ - 232a;^ + 35a;^ + 35a;^ - 16a;^ + 10a;^°)/ + (36 - 260a; 

+ 672a;^ - 806a;^ + 532a;* - 232a:^ + 64a;^ - lla;'' - 33a;^ - 10x^)y^ 

+ (-84 + 532x - 826a;2 + 5693;^ - 249a;* + 35a;^ - Ua;^ + 48a;^ + 4a;^)/ 
+ (126 - 686.x + 588x2 _ -^^g^a ^ ^-^q^4 ^ 35^5 _ 33^6 _^ 43,7)^8 

+ 2(1 - x)(-63 + 224x + llOx^ + 65x^ + 13x* + 5x-'^)?;^ - 2(1 - x)(-42 
+ 112x + 84x2 ^ 322,3 _^ 52,4)^10 _^ _ ^'^(^_^q _^ 54^ ^ 42-^2 _^ 53.3^^11 

+ (l_^)2(9 + ^)yl2_(i_2;)yl3 
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(a) The top factor extends far- (b) The indent factor extends 

thest to the right. farthest to the right. 



Figure 14: The form of a bimodal 2-unimodal polygon with the indent on the 
bottom side. 

6 Enumerating 2-unimodal polygons 

6.1 Bimodal 2-unimodal polygons 
6.1.1 Indent on the left 

Following Section 3.1 mutatis mutandis, we obtain the following generating func- 
tion by distinguishing a vertical step: 



6.1.2 Indent in corner 

Following Section 4.1.2 mutatis mutandis, the generating function for bimodal 2- 
unimodal polygons with their indent in the corner, such that the bottom factor is 
further to the right than the indent factor, is2 SP"^ Z {Z —1/ (l—u)) / {l—u) . When 
the indent factor is further to the right, the generating function of the indent 
factor for base g is X;3>i ^(^'(l - a;)' + 83(1 - x^) + 2(1 + 4x-|-a;2))x^^/6(l - x)^, 
whereas the other factors are enumerated straight-forwardly as v 
Summation then gives the required result. 

6.1.3 Indent on bottom 

If the indent of a bimodal 2-unimodal polygon is on the bottom, its form is one 
of the two shown in Figure 14. The case where the top factor extends farthest 
to the right is given by the inclusion-exclusion method as 



If the indent is farthest to the right, we note that it is slightly different to 
the case when the indent is in the corner, as we have already counted the case 




5 f UP — xyj (1 — x) — x(l + x)y 



^l(\-xf 
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(a) The left-side indent is above the corner 
indent. 




(c) The indents can be put at the same 
height. Then, the left indent can be moved 
into a position determined analytically by 
taking the derivative. 



(b) Both indents are in the corner. 




(d) The indent factor extends farthest to 
the right. 



Figure 15: The form of a 2-unimodal polygon with its indents on the top. 



where it is a pyramid. The generating function is 



E 

g,h>l 



x^y \ x{g^{l - x)2 + 2(1 + x) + g{b - 6x + x^)) 



1-xJ 2(1 



xy I x \ j 1 ^ 



l-x-y\l-yj \l-y 



6.2 Case 1: indents in the same direction, on the same edge 
6.2.1 Indents on top 

We enumerated the case where m-unimodal polygons have all their vertical 
indents on the left side in Section 3.1. The case where the indents are at the 
same height follow from Section 4 mutatis mutandis. We therefore have two cases 
remaining to enumerate: when one indent is in the corner and the other is either 
on the left or in the corner as well. Their form is shown in Figure 15. We combine 
parts (a), (c) and (d) of the figure and enumerate it by distinguishing a height 
of a 1-unimodal polygon with its indent in the corner, giving the generating 
function 

5 / „ 1 



ly'—ISPZ {uZ+ ^ 

oy \ 1 — X 
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Figure 16: The form of a 2-unimodal polygon with both indents on the bottom. 



The case where both indents are in the corner can be generated by taking 
the derivative with respect to height of only the pyramid factor, or simply by 
inspection, due its corner-staircase form. Its generating function is 



6.2.2 Indents on bottom 

If both indents are on the bottom, then the polygon is of the form illustrated 
in Figure 16. The generating function follows from Sections 3 and 4.1.3 mutatis 
mutandis. 

6.3 Case 2: indents in the same direction, on different edges 

When the indents are both vertical and on different edges of the polygon, the 
top indent is either on the left edge, or in the corner. They therefore belong to 
one of the cases depicted in Figure 17. 

6.3.1 Top indent on the left 

Let us first consider the case where the top indent is on the left edge. This case 
is given in Figure 17(a), which one can construct by joining the middle factor, 
enumerated by T{s, t), with the top one. It is clear, from Figure 17(b), that when 
the middle factor is farthest to the right, such that fixed, joined steps of the top 
factor form a 1-dimensional loop and the middle factor is wrapped, we may well 
form a 1-dimensional loop with the steps along its base. Now, as was shown in 
Section 3.2, moving the last vertical step down to the right-most edge preserves 
the self-avoiding condition. However, we will now need to enumerate the base 
according to a certain parameter, say t, and when the bottom horizontal steps 
change direction and form a 1-dimensional loop along the bottom, the length of 
the base is double-counted. We must therefore enumerate these configurations 
according to the length of this 1-dimensional loop, which becomes the height 
one row shown in the figure, in order to adjust for this. This can be done by 
taking the derivative of the pyramid generating function to distinguish a height 
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(a) The bottom indent is below the 
top indent. 




(c) The indent factor is farthest to the 
right. 




(e) The level indents are interweaved. 



(b) The middle factor is further to the 
right than the top factor. 




(d) The bottom indent is level 
with the top indent. 




(f) Interweaved indents have a 
pyramid top factor. 



(g) The bottom indent is above the top (h) The top indent is in the cor- 

indent. ner. 



Figure 17: The form of a 2-unimodal polygon with indents on opposite edges. 
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at which to place the indent. We then remove the right-most height one row that 
has weight xt for each cell, and replace it by a row of at least length one that 
weights its cells by xrt, such that r counts the right-end length. The generating 
function is therefore 

xrt{l-xt) ( xty Y S P{xt,y) ^^^^ 



1 — xrt \1 — xt J Sy y 

We therefore generate the configurations shown in Figure 17(a), using the inclusion- 
exclusion principle, with the function 



E 



.^11 y. I ll \ ."s 

QsT{s,t) 



y X 



xy" s sv 2xy\ 



I- s-v A y (1 - s)3 



xH^y'^ ft 1 \ \ t^v 

®t ,,,,, , (31) 



((l-a;t)2-y)2 \^l-a;t2 I - xt J ) {l-tf{l-t-vy 

Now, if the indent factor is farthest to the right, it is of the form shown in 
Figure 17(c). This form is very similar to the adjustment we had to make in the 
previous enumeration and its generating function follows mutatis mutandis. 

The case in which the indents are side by side is shown in Figure 17(d), (e) 
and (f). These are straight-forward to enumerate using our standard techniques, 
as is part (g), which depicts the case where the bottom indent is higher than 
the top indent. 



6.3.2 Top indent in the corner 

Let us now consider the case where the top indent is in the corner. Its form is 
shown in Figure 17(h), where it can be seen to be similar to the form of part 

(a) of the figure. Due to this symmetry, we must ensure that the top, pyramid 
factor is of at least height two. Its generating function then follows from the 
enumeration of parts (a) to (c) of tlie figure mutatis mutandis. 



6.4 Case 3: indents in different directions, on the same edge 

We now consider 2-unimodal polygons that have indents in different directions 
on the same edge of the polygon. We note that this means that they are either on 
the same side, or one is in the corner and the other is not. Now, the cases where 
one indent is on the left are symmetrical, up to an interchange of variables, with 
cases with one indent on the bottom. We therefore only consider those cases 
where there is an indent on the left, or they are both in the corner. 

We start by considering the two cases where they are on the same side. There 
are two distinct cases: when both indents are on the left edge and when they 
are in the corner. In either case, when the vertical indent is below the horizontal 
one, the polygon is locally concave around the indent, in the Euclidean sense. 
For example, see Figures 18(a) and (c). Otherwise the polygon is locally convex 
around the indents, as in Figure 18(b) and (d). 

The remaining polygons have indents on adjacent sides. The only cases we 
have to consider therefore have one indent on the left edge and one in the corner. 
These are: when the left indent is vertical and when it is horizontal. 
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We note that if a vertical (resp. horizontal) indent touches the top (resp. 
right) edge of the minimum bounding rectangle, it could be considered to be on 
the left (resp. bottom) edge or in the corner. We arbitrarily choose to include 
such polygons in the case where the indents are on different sides. 



6.4.1 Indents on the left edge 



Locally concave. When the two indents form a concave edge, they are of the 
form shown in the first diagram of Figure 18(a). We note that they must have 
a top factor of width at least two, giving the generating function 



ySP^Z E 



2 * 
X U 



2xSP 



y){x — u*Y\ 



There is one sub-class of polygon whose indents intersect, whose form is shown 
in the second diagram. These are enumerated by 



yuSPE 



.{^ - y)i^ - X - y) \{x - u*) 1 



Locally convex. When the two indents form a convex edge, they are of the 
form shown in Figure r7(b). Again, there is a subclass whose indents intersect 
given in the second diagram. We note that the vertical indent may not lie along 
the top of the minimum bounding rectangle, and so the top factor must be of 
height greater than one. The generating function is given by standard methods. 



6.4.2 Indents in the corner 

Locally concave. In this case, the horizontal indent is above the vertical one. 
And so, if the indents are distinct, as in the left diagram of Figure 18(c), the 
top factor is a pyramid with a horizontal indent. This is therefore enumerated 
by P'{u,y), given by equation (24), and the overall generating function is 

I.p'{u,y). (^SPZ+j^y (32) 

which can be re-expressed in the following form that is symmetric in x and y 
to reflect its geometrical symmetry: (1 + u){l + v)SP^Z^/{l — x){l — y). Oth- 
erwise, there is a 2-dimensional indent, as in the right diagram in the flgure. 
Noting that the vertical (resp. horizontal) indent factor can extend further to 
the right (resp. top) than the bottom (resp. top) factor, the generating function 
for this case is 

X • X(l - (1 - u){l - v)) ( SPZ + ^) (^SPZ + ^) . 

Locally convex. There are no complications in this case. We again note that 
the indents may not lie along the edge of the minimum bounding rectangle. 
Thus, the generating function can be given on inspection of Figure 18(d), and 
simplified to {u + v) SP^Z^/xy. 
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(a) The indents form a concave region on the left edge. 




(b) The indents form a convex region on the left edge. 




(c) The indents form a concave region in the corner. 




(d) The indents form a convex region in the corner. 



Figure 18: The form of a 2-unimodal polygon with indents in opposite directions. 
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6.4.3 Horizontal indent on left, vertical indent in corner 

When there is a horizontal indent on the left edge, and a vertical one in the 
comer, the polygons are of the form depicted in Figure 19. They are divided 
into four classes, according to the position of the horizontal indent relative to 
the vertical indent: above, next-to, adjacent and below. We say that the indents 
are next to each other if the indentations formed overlap in height. If the hump 
of the indent factor (which is the non-convex part of the interior of the polygon) 
is at the same height as the vertical indent, we say that they are adjacent. 

When the horizontal indent is above the one in the corner, we see that 
reflecting the top, pyramid factor would give the case where both indents are 
in the corner, such that the polygon is locally concave around the indents. This 
implies that the indents are distinct, as in the first example of Figure 18(c), 
which is enumerated by ZZySPZ{l/{l — x) + uZ)(SP/{l — y) + uZ). 

When the horizontal indent is next to or below the one in the corner, the 
generating function is obtained simply by standard methods. This is also true for 
when the horizontal indent factor is adjacent to the vertical indent, except that 
we note that the pyramid factor in the first quadrant can be empty, as shown 
in the second diagram of Figure 19(c). In this case, the generating function is 
given by 

s ( I ^ \ sy ( y ^ (1-*)' 



1 — s J \1 — X X — J 1 — s — y \l — y (1 — s)"^ — y 

xsty ut^ 
Qs z ^ &t 



+ 



1-xs-ty (l-t)2(l-n-t) 
s \ I 1 s \ y 



1 — s J \1 — X X — 1 1 



xsty ut"^ 
Qs r, —— Qt 



l-xs-ty (1 -t)2(l -M-i)' 

6.4.4 Vertical indent on left, horizontal indent in corner 

The class of polygons that have a vertical indent on the left edge and a horizontal 
one in the corner, as shown in Figure 20(d), is equivalent to the second case of 
Figure 20(a), after one flips the right factor and the horizontal indent. However, 
there are three differences: 

• we must allow the possibility of the indent touching the top of the mini- 
mum bounding rectangle, such that it is at least as high as the left factor; 

• when the indent is the highest factor and intersects itself, as in Fig- 
ure 20(e), a vertical indent can be placed next to the width one indent 
factor (the "hump") in the top right, and the resulting polygon will be 
self- avoiding; and 

• the right, pyramid factor must be of at least height two, for when the 
indent lies along the right edge, we have chosen to consider this to be on 
the bottom edge, rather than in the corner. 
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(a) The left indent is above the one in the (b) The left indent is next to the verti- 

corner. cal one. 




(c) The indent factor is adjacent to the vertical indent. 











1 1 











(d) The left indent is below the other. 



Figure 19: The form of 2-unimodal polygons with a horizontal indent on the 
left, and a vertical one in the corner. 
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(a) A horizontal indent on the bottom. 



(b) The symmetrical class. 




(d) A horizontal indent in the cor- 
ner. 



(c) The exclusion case of the sym- 
metrical class. 




(e) The indent factor intersects it- 
self. 



Figure 20: The form of 2-unimodal polygons with a vertical indent on the left, 
and a horizontal one on a different edge. 
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The first point adds a 1/(1 — y) term to the expression, and the last adds a 
multiphcative factor ofu/{l — v) to the contribution of the pyramid. The second 

case is separate. Taking the derivate of the indent factor gives ^| • jjziy^^ = 

y^/(l— y)^. This leaves us with the following expression for the overall generating 
function: 

^ oSf SP^Z u 

^'y ^\ ~^ — 

oy \ xy 1 — V 

6.5 Case 4: indents in different directions, on opposite edges 

When the indents of a 2-unimodal polygon are on opposite edges, there is one 
on the left and one on the bottom. There are therefore two cases to consider: 
when the left indent is vertical and when it is horizontal. 

Before starting, we note that when we enumerate those cases where the 
indent lies along the MBR there is one case that has already been enumerated 
in the previous case: when the vertical indent is on the left perimeter, and the 
horizontal indent lies on the right perimeter. 



6.5.1 Vertical indent on left, horizontal indent on bottom 

When we have a vertical indent on the left and a horizontal indent on the 
bottom, an exchange of the variables x and y (corresponding to a reflection in 
the X = y axis) is an automorphism, giving the same class of polygons. Now, if 
the vertical indent is to the left of the horizontal one, as depicted in Figure 20(a), 
there is a staircase factor to the bottom-left, allowing us to place the indent by 
taking the derivative. We can obtain those with their horizontal indent below 
the vertical one via an exchange of x and y in the generating function. We can 
then obtain those polygons whose vertical indent is not to the left by excluding 
the double-counted, symmetrical case, shown in Figure 20(b), whose vertical 
indent is also to the left. And so, we only have to enumerate these two classes. 

Considering those polygons of the form given in Figure 20(a), we can place 
the indent distinguishing a vertical step in the left factor. We must therefore 
separate the two cases shown (according to whether the left or right factor is 
highest), as the usual bijection which allows us to rejoin these cases does not 
distinguish between the contribution to the height from the two factors. The 
first case, when the right factor is highest, is enumerated by 



x^(l — x)^ 1 2xy uv* 

5y* \y* 



- X — V* (1 — x)'^ ~ y\ ^ 1 — u — V 

where the derivative is taken to act only on those factors that are asterisked, 
and afterward y* is equated with y. The case where the right indent is higher is 
simpler, as there is no contribution to the height from the right factor. We note 
that one can flip the right factor to obtain the form shown in Figure 20(d), whose 
generating function one can write on inspection. Finally, the symmetric case, 
shown in Figure 20(b), is enumerated using the inclusion-exclusion principle. 
We then note that this enumerates polygons whose fixed steps along the base 
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of the top-right factor form a 1-dimensional loop, as shown in Figure 20(c), 
which we must exclude. Similarly, the fixed steps along the left edge may form 
a vertical 1-dimensional loop. Surprisingly, the generating function simpliies to 
— x — y)/A^/^, which is symmetrical in x and y. Thus, the exclusion case for 
both the horizontal and vertical 1-dimensional loop have the same generating 
function. Finally, the doubly-excluded case, of both a horizontal and vertical 
1-dimensional loop, is re-included to give the required result. 

6.5.2 Horizontal indent on left, vertical indent on bottom 

When the indents of a 2-unimodal polygon consist of a horizontal one on the 
left edge and a vertical one on the bottom, the polygon is of a form depicted in 
Figure 21. Again, these forms divide the polygons into four parts according to 
the height of the left indent, relative to that of the vertical indent. 

When the horizontal indent is above the vertical one, the latter indent could 
be to the right, under, adjacent to, or to the left of the horizontal indent above it. 
However, similarly to the previous case, the latter three sub-cases are symmetric 
with other cases, and we need not enumerate them. 

All of the following four cases can be enumerated by an inclusion-exclusion 
argument, together with our other standard techniques. We note that we have 
to exclude the polygons whose fixed steps along the factorisation lines form a 

1- dimensional loop, as in the second diagram of part (a) of the figure. We also 
note that we must enumerate the cases where the indent factors are farthest to 
the right or the top. 

6.6 The 2-unimodal generating function 

Combining all of the above cases gives the following generating function for 

2- unimodal polygons: 



2x3y3(l_ar)3(l-y)3A5/2 

r{") 

^2 /ooN 

2x3y3(i _ a;)5(i _ y)5((i _ a,)2 _ y)3((i _ y)2 _ a.)3(i - x - y)' ^ ' 

where A^"^ = 2(1 - x^'^x'^ - 2(1 - x)^x'^il3 - lOx + 3x^)y + 2(1 - x^x'^iA - lOx + 

77x^ - 97x^ + m-r" - 23.r'^ + 3x%f - 2(1 - xfx'^{38 - 109x + 328x'^ - 501x^ + 314a;4 - 
113a;5 - 5x6 _ ^^^7 _^ ^s-^yS _^ _ ^-,4(2 _ ig^ + 'd78x^ - 878x^ + 16563;^ - 2024x^ + 
lU5x^ - 294x^ -I- 45x^ + 122x^ + 2xi°)y^ - (1 - xfi22 - 188x + \M2x^ - 2610a;3 -|- 
3402.x4 - 3350x5 ^ 1832x6 - 337x^ + \9i2x^ - 159x9 - 40x^0 + 2x")y5 + (1 - x)(110 - 
1012x 4640x2 - 10118x3 12949x'' - 11749x5 7989x6 _ 33942.7 1773,8 _^ 213x9 ^ 
175x1° - 120x^1 -I- 12xi2)y6 ^ (•_33o + 2958x- 10938x2-1-21112x3 - 24520x^-1-19285x5 - 
11383x6 5556.x^ - 1584.x* - GlLx" + 555xi" - 160x^1 + 28x^2)^7 _^ (ggg _ 5172^; + 
14664x2 - 20918x3 ^ 17843x4 - 10049x5 ^ 3571x6 - 1584x^ 1342xS - 295x9 82x1° - 
32x^1)?/^ -I- (-924 -h 6300x - 13980x2 + 14186x3 - 8426x^ -|- 3578x5 ^ 2,%y& _ ^^^^1 _ 
295x8 - 24.t9 + 14xi0)?y9 + (924 - 5460x + 9708.x2 - 6454.x3 + 2105.x4 - 1350x5 - 38x6 _,. 
555x^ 82x*^ 14x9)yi" + (-660 + 3372x - 5012x2 ^ 1968x3 ^ 250x'^ -I- 547x5 - 295x6 - 
160x^ - 32x*)j/ii -I- (330 - 1458x + 1930x2 - 496x3 - 431x^ - 33x5 _^ 1323,6 ^ 28x^)2/^2 _ 
2(1 - x)2(55 - lOlx -h 7x2 ^ 353.3 ^ 6x^)2/13 _ 2(1 - x)3(-ll -h 4x -h x^)y^^ - 2(1 - xfy^^ 
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1 1 




1 















(a) The left indent is above the one on the bottom. 




(b) The left indent is next to the verti- (c) The indent factor is adjacent to the 
cal one. vertical indent. 













J 






1 





(d) The left indent is below the other. 



Figure 21: The form of 2-unimodal polygons with a horizontal indent on the 
left, and a vertical one on the bottom. 
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and B)^' = -2(1 - xf^x^ + 2(1 - x)^^x'^{2Q - 26x + 5x^)y - 2(1 - xY^x^{4 - lOx + 
189x2 - 464a;3 + 388a;'' - 122a;5 + lOa;^)^^ + 2(1 - x^^x^ (66 - 243x + 1343a;2 - 4004a;-^ + 
5623a;^ - 4025x^ + U59x^ - 230x^ + 10x^)^3 - (1 - a;)i°(2 - 20a; + 1096a;2 - 5130a;-'' + 
17888a;'' - 49004^^ + 861 lla;*^ - 92423.2;'' + 60313.x8 - 23283a;9 + 4864a;i° - 440a;" + 
10a;i2)y4 + (1 _ a.)8(-3g _ 404^; + 6712a;2 - 34332a;3 + 107222a;'' - 257436a;5 + 471559a;6 - 
618724a;^ + 559981a;^ - 341505x^ + 135916a;i° - 33115a;" +4316a;i2 - 220a;^^ + 2x^^)y^ + 
(1 - a;)6(-306 + 3806x - 36028^^ + 177500a;3 - 546201a;4 + 1230135^5 - 2183847a;6 + 
3061206a;^-3288667a;H2624163a;9-1512420a;i° + 609513a;"-163267a;i2 + 26485a;i3- 
2144a;" + 52x^^)y^ - (1 - a;)*(-1632 + 22272x - 1703043;^ + 786188a;3 - 2405909a;* + 
5350296a;5 - 9271113^^ + 13015854.t^ - 14902848a;8 + 13692922a;9 - 9840757a;io + 
5379124a;" - 2166934a;i2 + 616413a;i3 - 115521x1" + 12519x^5 - 582x^6 + 4x")/+ 
(1 - x)3(-6120 + 84760X - 581646x2 + 2434626x3 - 6888332x" + 14198657x5 - 
22614758x6 + 29071458x^ - 30896121x*' + 27177494x9 - 19436717x^0 + 109871 13x" - 
4750238xi2+l510618xi'''-334368xi''+46862xi''^-3416xi6+72x")y^-(l-x)2(-17136+ 
240688X - 1579636;e2 + 6285816x-' - 16975856x4 + 33355749x5 - 50170572x6 + 
60137293x^ - 59249831x« + 48829663x'^ - 33627193x"' + 18983318x" - 8514558x^2 + 
2920248x1'* -729963x1" + 123709x15 -12407x16 + 524x1^)7/ -(1-x) (37128 -528528X + 
3424538x2 - 13385812x3 + 35510647x1 - 68394451x5 + 100101788x6 - 115262274x^ + 
107466746x« - 83069604x9 ^ 53987064x1° - 29441819xii + 13202121x12 - 4701667x1^ + 
1271236x14 - 245731x15 + 30752x16 - 2002xi^ + 28x^^)j/" + (63648 - 917696x + 
5965232x2 - 23360456x3 + 62122407x1 - 119859819x5 + 175071125x6 - 199526930x^ + 
181725867.7;«-135067367x9 + 83428883xi"-43315180xii + 18849138xi2-6726629xi3 + 
1892384xi4-397178xi5+57372xi6-4893xi^+152xi8+4xi9)yii + (-87516+1189760x- 
7206914x2 + 26101372x3 - 63794722x4 + 112416740x5 - 148870993x6 + 152402508x^ - 
123199222x8 + 80108387x9 - 42643940x1° + 18849138xii - 6931650x12 + 2077871x13 - 
484525x1" + 81618x15 - 8776x1° _^ 4453.17 _^ 2x^^)y^'^ + (97240 - 1241240x + 7002476x2 - 
23476728x3 + 52823226x1 - 85191730x5 + 102552835x6 - 94614843x^ + 68159388x8 - 
38932682x9 + 17903788x1° - 6726629xii + 2077871x12 - 519020x13 + 99499x1" - 
13256x15 + 991x16 - 16xi^ - 2x^^)y^^ + (-87516 + 1043900x- 5472038x2+ 16957822x3 - 
35081419x1 + 51716269x5 - 56520264x6 + 46940030x^ - 30104049x8 + 15085017x9 - 
5972903x1° + 1892384xii - 484525x12 + 99499x13 - 15652x1" + 1683xi5 - 86xi6)t/i4 + 
(63648 - 705536X + 3423376x2 - 9772052x3 + 18515467x" - 24837930x5 + 24516909x6 - 
18219941x^+10332058x8-4503883x9+1516967xi"-397178xii+81618xi2-13256xi3+ 
1683xi"-156xi5+6xi6)j/i5 + (-37128+380016x-1697802x2+4439228x3-7652806xi+ 
9266605x5 - 8180675x6 + 5376370x^ - 2657724x« + 989788x9 _ 276483x1° + 57372xii - 
8776x12 9912.13 _ gg^i4 ^ 6xi5)yi6 + (17136 - 160720x + 656508x2 _ 1559760^3 + 
2421888x1 - 2613261x5 + 2030050x6 - 1155943x^ + 485274x8 - 149047x9 + 32754xi° - 
4893x11 + 446x12 - 16xi3)yi7 + (-6120 + 52120x - 192822x2 + 411534x3 - 566923xi + 
534144x5 - 355377x6 ^ i69i05x^ - 57326x8 + 13455x9 - 2030xi° + 152xii + 2xi2 - 
2xi3)yi8 + (1632 - 12480X + 41276x2 - 77826x3 + 92923xi - 73915x5 + 40129x6 - 
14871x^ + 3632x8 - 524x9 + 28xi° + 4xii )?/i9 - 2 (1 - x)" (153 - 424x + 392x2 - 155x3 + 
36xi)?/20 + 4(1 - x)5(9 - 8x + x'^)y^^ - 2(1 - x)5y22 

7 Enumerating 2-convex polygons 

There is a choice of side when the indent occurs at the edge, that is, touching the 
MBR. We choose to define this according to the relative position of the indents, 
such that they occur either on the same side, or the closest possible sides. This 
is relevant to Cases 3 and 4, but is in contrast to the choice made in Cases 3 and 
4 of the previous section. However, the choice is arbitrary and does not make a 
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Figure 22: The form of a 2-convex polygon with both indents on the same side. 

difference to the methodology. 

In our 'divide-and-conquer' approach, we factor the polygons along lines 
that run along the base of the indentations. When the indents are in different 
directions, we can therefore divide the plane into four quadrants, by extending 
these factorisation lines. The first quadrant is in the top-right, and they are 
ordered in an anti-clockwise fashion. 

7.1 Case 1: indents in the same direction, on the same edge 

If the indents of a 2-convex polygon are in the same direction and are on the 
same edge, then they can be further classified according to whether they are on 
the same side or not. We will assume without loss of generality that they are 
both vertical and on the top edge. 

If one or both of the indents lies along the top edge of the MBR, then they 
could be considered to be on either side of the polygon. We choose by default 
that the indents be classified as on different sides, where possible, as this will 
force us to make an adjustment to the easier of the two cases. Moreover, when 
the indents lie on the same side, there is a symmetrical case (e.g. when the 
indents are on the top-left and top-right sides) for which this choice avoids 
double-counting. 

7.1.1 Indents on the same side 

If the indents of a 2-convex polygon are both on the same side, then they are 
either at the same height, or they are not. Due to our choice that the indents 
be considered on different sides where possible, the top factor must be of at 
least height two. We refer to Figure 22 for the form of these polygons, which 
is the same as the unimodal case, except that the bottom factor can now be 
unimodal, not just staircase. We enumerate the top factor as unimodal, and use 
wrapping to include the case where the middle or bottom factor touches the 
right edge of the MBR. This is a perfect example of how versatile wrapping is 
- there are even exclusion cases that intersect in the bottom right corner, with 
a 2-unimodal self-avoiding factor that has both indents in the corner. 
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We adopt an inclusion-exclusion argument, allowing the unimodal factors 
to intersect, and then exclude the ones that factor as a 2-unimodal intersecting 
polygon and a unimodal SAP. Noting that we must now have a top factor of at 
least height two, we obtain these generating functions from Section 6.2 mutatis 
mutandis. 



7.1.2 Indents on adjacent sides 

When the two vertical indents of a 2-convex polygon are both on top, but on 
adjacent sides, their form is very similar to the previous case, where the indents 
are on the same side. As outlined in the unimodal case 1 (c.f. Figure 15), once we 
wrap the polygon such that the middle factor is the one that touches the right 
edge of the MBR and the top factor is a pyramid, we can take thse polygons and 
flip top factor and indent (that is, along the top join) such that the indents are 
on adjacent sides (c.f. Figure 15(d)). In this case the top-right indent is above 
the top-left one. Similarly, if the flip is done along the bottom join, we obtain 
the case where the top-right indent is below the top-left one (c.f. Figure 15(a)). 
Obviously, now that the polygon is not rooted in one corner, these two cases 
are symmetrical. The one difference in this case is that now the top indent can 
extend furthest to the right, and may touch the right edge of the MBR. We 
therefore need to include this case as well. 

Proceeding as above, mutatis mutandis, the generating function when the 
indents are at different heights is therefore 
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and the case where the indents are at the same height is obtained similarly. 



7.1.3 The generating function. 

The above two generating functions combine, with a multiple of two for the 
symmetry of the indents being on the same side, to give the generating function 
for 2-convex polygons with both vertical indents on top. This result then gives 
the overall generating function 

2A{x,y) 



xV(l - a;)2(l - 2/)2((l - y)2 - x)((l - x)2 - y)A7/2 

+ 2i?(x,y) 

^ X2y2(l _ ar)2(l - j;)2((l _ y)2 _ 3.)2((l _ _ y^2/^4 ' ^''V 

where A{x, y) and B{x, y) can be found in [14]. We note that A{x, y) is 0{x^'^y^'^), 
A{x,x) is O(x20), B{x,y) is 0{x'^^y'^^), B{x,x) is 0{x^^). 
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(a) The middle and top factors touch (b) The middle factor touches both 

the left and right edges respectively. the left and the right edges. 




(c) The top indent touches the left edge (d) The top indent touches the left edge 
and the top factor touches the right. and the middle factor touches the left. 



Figure 23: The form of 2-convex polygons with indents on opposite sides. 

7.2 Case 2: indents in the same direction, on opposite edges 

The case where the indents are in the same direction, but on opposite edges, 
is equivalent to Case 2 of the 2-unimodal polygon enumeration of the previous 
section, whose form is depicted in Figure 17. Now the polygon is no longer 
rooted in the bottom left corner, and so we break up the enumeration into two 
parts that are equivalent to those of the last section. 

7.2.1 Indents on opposite sides 

When the indents of a 2-convex polygon are on opposite sides, its form is as 
depicted in Figures 17(a)-(e) and (g), except that the bottom left factor is 
unimodal. We note that the form shown in part (f ) of the figure has the indents 
on opposite edges, but adjacent sides, and does not belong to this case. When 
the bottom indent is below the top indent, the form is as shown in part (a) 
of the figure. Proceeding as per Section 6.3, we join the top unimodal factor 
and indent to the middle staircase factor, wrapping the resulting polygon such 
that the middle factor may touch the right edge of the MBR. However, again, 
when the base of the staircase factor is folded, as in part (b) of the figure, 
an adjustment must be made. The top two thirds of the polygon is therefore 
enumerated as in equation (31). 
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What is different about the convex case is that we cannot simply apply the 
same logic to join the bottom factor as before, when we could adjust for the 
folding of the base before applying the join. Now, if we intend adding a unimodal 
factor to the bottom that can be wrapped, such that it becomes a pyramid, we 
must then remove the cases where the top indent is folded. Consideration of this 
case will reveal that the top join requires that the top-right unimodal factor must 
not be folded, or the join will have a factor s/{x—s) on one side, and s'^x/ (1— s^x) 
on the other, which causes the result to diverge. Enumerating the cases that 
were enumerated by the folded factor separately makes the enumeration just as 
complicated as if we never folded the bottom factor in the first place! And so 
we don't, but instead enumerate the two cases determined by whether or not 
the middle factor touches the left edge of the MBR. 

We therefore proceed by enumerating the cases where the bottom-left uni- 
modal factor touches the left edge of the MBR by following the unimodal cases 
in Section 6.3 mutatis mutandis. The case where it is the middle factor that 
touches the left edge can be divided into two parts, as shown in Figure 23(a) 
and (b), which are also generated by standard techniques. We note that the 
latter case is the first instance where we require the convex generating function 
from Section 3.4.3 that is enumerated according to both its top and bottom 
perimeter. We also need to generate the symmetrical cases where either indent 
(but not both) touches the side. Their form is shown in Figures 23(c) and (d). 

7.2.2 Indents on adjacent sides 

The second part of the enumeration of 2-convex polygons with their indents 
on opposite edges is when they are on adjacent sides. This is equivalent to the 
unimodal cases depicted in Figures 17(f) and (h). We enumerate them by taking 
the previous classes with indents on opposite sides, and flipping the bottom 
factor and indent. 

Now, let us consider the opposite-side cases shown in Figure 23. It is evident 
that if the bottom factor and indent are reflected such that the indent is to 
the left, it will now be on the adjacent side. Furthermore, we notice that parts 
(c) and (d) of the figure show when the top indent touches the left-edge of the 
MBR. If we have swapped the bottom factor and indent in these cases, it could 
be the bottom factor that touched the right edge of the MBR. There is therefore 
an extra case to enumerate. 

There is another fact that we must note - that if the top or bottom factors 
are of height one, then they were enumerated above, as the indent could be 
considered to be on either of two sides. We must therefore re-do the enumeration 
of the polygons of Figure 23, but with factors of minimum height two. (We note 
that it is helpful to first enumerate the extra cases with factors of minimum 
height one and then combine them with the results from above to verify the 
total against series which include the double-counted polygons.) 

We finish by noting that the interweaved case, equivalent to the unimodal 
polygons shown in Figure 17(f), does not follow as simply from the unimodal 
enumeration as the other cases. By making a unimodal factor wrap a pyramid 
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bottom factor, we obtain the generating function 
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7.2.3 The generating function. 

Summing the above two results gives the generating function for polygons with 
two vertical indents on opposite edges, with one being in the top-left corner. To 
obtain the generating function for all those with two vertical indents on opposite 
edge we multiply by two, due to symmetry. However, we are now double-counting 
the case where the top and bottom indents touch the top and bottom of the 
MBR respectively. Denoting = (d/dx)"' /n\, this case is generated by 

At — 
^t'^t {stC{s, t)) + Y^'i'l^t {tU{s, t)) 

Then, a change of variables gives the generating function for two horizontal 
indents, which in turn gives the following generating function for 2-convex poly- 
gons with indents on opposite edges: 

~2A{x,y) 



xV(l-a;)3(l-y)3AV2 



B{x,y) 

where A{x, y) and B{x, y) can be found in [14]. We note that A{x, y) is 0{x^^y^^), 
A{x,x) is 0{x^^), B{x,y) is 0{x'^^y^^), B{x,x) is 0{x^^). 



7.3 Case 3: indents in different directions, on the same edge 

We now consider two out of three cases which enumerate 2-convex polygons 
with indents in different directions: where the indents are on the same side and 
where they are on adjacent sides. This is equivalent to Case 3 of the 2-unimodal 
polygon enumeration of the previous section. When they are on the same side, 
they form a locally concave or convex region, as per Figures 18(c) and (d) 
respectively. When they are on adjacent sides, their form is equivalent to that 
shown in Figure 19. 



7.3.1 Locally concave 

If the polygon's indents form a locally concave region, they are of the form 

depicted in Figure 18(c). Aside from the walk between the two indents (which 
is in the first quadrant by definition) the polygon can either enter the first or 
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the third quadrant, but not both. Those polygons which do not enter either 
quadrant, passing through the origin, we will consider to be passing through 
the first quadrant. 



Third quadrant. Let us first consider those polygons that enter the third 
(bottom-left) quadrant. The indents are both in the first quadrant, within which 
the polygon is enumerated by Qi{s,t) = .s^t^(s + t)/(l — s){l — t){l — s — t). 
Considering the factor in the second quadrant, we define 



Q2{x,y;s,t) 
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and, by symmetry, Q,i{x,y: s,t) = Q2{y, x;t, s). And so, as the factor in the 
third quadrant is simply a directed walk, we have the following inclusion case: 
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Wrapping then forms all of the required polygons, including those cases where 
the polygon is unimodal in the thrid quadrant. The polygon may therefore in- 
tersect in any of the second, third or fourth quadrant, forming unimodal loops. 
Completing the inclusion-exclusion argument by removing those intersecting 
polygons, enumerated as per the incusion case mutatis mutandis, gives the de- 
sired result. 



First quadrant. We will now consider those polygons that pass through the 
origin, or enter the first quadrant. Aside from the indents, the factor in the 
first quadrant is staircase. We must join unimodal factors to the bottom and 
to the left, unless the indent factors touch the MBR, in which case, the second 
and fourth quadrant factors must be pyramids. As the unimodal factors can 
intersect themselves in the second and fourth quadrants, we adopt the usual 
inclusion-exclusion argument, and the generating function follows via standard 
techniques. 



7.3.2 Locally convex 

When the indents form a convex region, the 'humps' of the indents are either 
distinct, in which case the polygon does not enter the first quadrant, or they 
are joined by a common cell contained in the first quadrant. In the first case, 
the third quadrant factor is staircase, and otherwise it is formed simply by a 
directed walk. To this we join two unimodal factors in the second and fourth 
quadrant. The unimodal factors may intersect, and we use a standard inclusion- 
exclusion approach. 'Wrapping' the factors together ensures that all possible 
configurations of the polygons are formed. This means that despite the fact the 
factor in the third quadrant is staircase, when it is wrapped, it may intersect 
itself, forming a unimodal loop in the bottom-left corner, we therefore also need 
to exclude this case. 
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To help enumerate the unimodal factors of the second and fourth quadrant, 

let 
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and we define Q4 and similarly, with x and y interchanged. The inclusion 
case is therefore enumerated by 
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and the other terms follow mutatis mutandis. 



7.3.3 The generating function for indents on the same side. 

Summing the above two results for the polygons that are locally concave and 
convex around their indents gives tlie following generating function for polygons 
with indents in different directions on the same side: 

xyA{x,y) xyB{x,y) 



2(l-x)(l-y)A7/2 2(l-x)2(l-y)2A4' 

where A(x, y) and B{x, y) can be found in [14]. We note that A{x, y) is 0{x^y^ 
A{x,x) is 0{x^), B{x,y) is 0(a;^V°), B{x,x) is 0{x^^). 



7.3.4 Indents on adjacent sides 

We enumerate this case by braking it up into parts that are classified by the 
relative height of the indent. As per the previous section, we say that the indents 
are next to each other if the indentations formed overlap in height. If the hump 
of the indent factor, rather than the indentation, is at the same height as the 
vertical indent, we say that the)' are adjacent. And so, the horizontal indent is 
either above, next to, adjacent to or below the vertical indent. The form of the 
equivalent unimodal polygons are shown in Figure 21. However, there is one case 
missing that was not relevant for the unimodal situation; when the horizontal 
indent is above the vertical one, and the polygon touches the MBR in the second 
quadrant. This case is depicted in Figure 24. 

When the horizontal indent is higher, the polygons are of one of the forms 
shown in Figures 19(a) or 24. Let us first enumerate the latter case, which has 
a significantly different form to the other polygons of this class. We factorise 
as usual, which divides the polygon into two main factors: the top and bottom 
sections. 

We begin by enumerating the top section. We need to enumerate it according 
the its base. We have not already needed to enumerate 1-unimodal polygons 
whose indent is in the corner according to their base. And so, we break this 
section of the polygon into two parts again by extending a line from the bottom 



45 



Figure 24: The form of 2-convex polygons with indents in different directions on 
adjacent sides of the polygon and which touch the left edge of the MBR in the 
second quadrant. 

of the indentation (shown in red in the figure). We therefore join an almost- 
pyramid polygon to a unimodal polygon and obtain the generating function 

where s counts the base. We then join the indent factor by multiplying by 
(xs/(l — xs))^ and join it to the bottom section. 

Now the bottom section is unimodal, but if it is pyramid, the indent factor 
can extend further to the right. We therefore add the pyramid and non- pyramid 
factors separately. We do the join in the most straight-forward manner by over- 
lapping one row of the top and bottom factors (indicated by the shaded region 
in the figure), which requires a division of the generating function by y. In this 
case, we run into the case where the polygon intersects itself. We must therefore 
exclude this intersecting case. The generating function is therefore given by 

i (C/f (.) - Uai)) (y^) ' U{s/x) + P{s, y) - l) , (40) 

where U{s) is the unimodal polygon generating function, with s counting the 
bottom perimeter, and -P(x, y) is the pyramid generating function. 

If the horizontal indent occurs above the vertical one, but the polygon 
touches the left edge of the MBR in the third quadrant, its form is as depicted 
in Figure 19(a), except that the bottom factor is convex. The case where the left 
perimeter is in the second quadrant is obtained via wrapping. The remaining 
three cases are all obtained by a similar extension of the unimodal case mutatis 
mutandis. 

7.3.5 The generating function. 

In the above derivation, we have taken the position and direction of one in- 
dent, and then assumed that the other indent is one the next side, following 
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Figure 25: The form of 2-convex polygons with indents in different directions on 
opposite sides of the polygon that do not enter the third quadrant. 

the polygon in one of the two possible orientations. However, those polygons 
whose indent is on the other adjacent side also belong to this class. We obtain 
their generating function by swapping the variables x and y. And so, the total 
generating function is 



(l-x)(l-y)A5/2 (l_^)(l_y)((l_^)2_y)((l_y)2_^)A4' ^ > 

where A(x^ y) and B{x, y) can be found in [14]. We note that A{x, y) is 0{x^y^), 
A{x,x) is O(x^), B{x,y) is 0{x^^y^^), B{x,x) is 0{x^^). 

7.4 Case 4: indents in different directions, on opposite edges 

We note that if the indents are on opposite edges, they must be on opposite sides, 
due to symmetry. We again enumerate this case by braking it up into parts that 
are classified by the relative height of the indent, adopting the same terminology. 
The form of the equivalent unimodal polygons are shown in Figure 21. The only 
difference is that the factor in the third quadrant can now be unimodal. We 
note, however, that the indents must not lie along the edge of the MBR, as 
these polygons have already been enumerated, in contrast to the unimodal case. 

As was the case when the indents were on adjacent sides, there is one new 
case to enumerate. This is when the polygon does not enter the thrid quadrant, 
as shown in Figure 25. This is an interesting case, as we cannot proceed as nor- 
mal, which is because the horizontal and vertical joins (over s and t respectively) 
must be done simultaneously'^. We are forced to re-evaluate our approach and 
come up with a new way to evaluate such constructions. Using wrapping, we 
can make a unimodal factor on top by making the top factor staircase and add 
double-bonds to the bottom factor, which wrap the staircase factor. Each bond 
has two steps, contributing to the weight of the join, as well as a weight of 
1/x. This leaves us with the following expression for the bottom factor in the 

■^For an illustration of why this is, see [14]. 
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where the two terms in the last factor correspond respectively to the cases where 
the indent touches the right-edge and where only the convex factor touches it. 

Up until now it has been possible to do joins with such factors in them by 
simplifying other terms and applying the equivalent operations to them. Now 
we will have two distinct factors that both have expressions of this form, and so 
we cannot avoid evaluating it. We therefore need to re-express these terms in a 
form which we know how to evaluate, which we achieve by noting that 
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(42) 



This therefore achieves the required wrapping by only applying the E operator 
to the 1-dimensional loop part of the join, and not the indent factor. 

We now apply this approach to our current problem by joining the indents 
and pyramid factors to a staircase polygon that is enumerated by its bottom 
and side perimeters. 'Folding' the staircase factor twice creates a convex poly- 
gon, gives the possibility of intersecting itself. We will therefore exclude this 
case, which is straight-forward to enumerate. Keeping in mind that the pyra- 
mid factors must be of at least height two, we obtain the following generating 
function: 
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The reamining four cases are equivalent to those depicted in Figure 21. Using 
the above application of wrapping next to the indents, we obtain the generating 
functions by enumerating the first quadrant as a directed walk. We note that 

the unimodal factor in the third quadrant must be of at least width two, and 
of at least height two when the top indent is above the bottom one. The latter 
case is generated by 
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for minimum height and width one, and the required result follows the exclusion 
of the height or width one cases. The remaining three results follow mutatis 
mutandis, and we refer to [14] for the details of the calculations. 
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7.4.1 The generating function. 



The same symmetries in the x = y axis exist for the 2-convex case as for the 
2-unimodal one, and it is necessary to reflect the last three, non-symmetric parts 
and add them to the other classes to get the all possible 2-convex polygons with 
a vertical indent on the top-left side of the polygon and a horizontal one on the 
opposite side. Summation gives the following generating function: 

2(l-x)(l-y)AV2 

Bi^^y) (45) 



2(1 - X - y)(l - x)^l - y)2((l - x)2 - j/)2((l - y)2 _ a:)2A4' 

where A{x, y) and B{x, y) can be found in [14]. We note that A{x, y) is 0{x^^y'^^) 
A{x,x) is 0(a;"), B{x,y) is 0{x^^y^^), B{x,x) is 0{x'^^). 



7.5 The 2-convex generating function 

In the above derivations, the direction and position of one of the indents was 
chosen arbitrarily, such that the direction and position of the other determined 
which sub-class the polygon belongs to. Therefore, when the two indents are 
in different directions, the cardinality of the set of possible combinations of 
direction and location for the fixed indent is four. When the indents are in the 
same direction, both directions are enumerated by the generating functions. If 
the indents are on the same edge, the cardinality is two. If they are opposite, 
it is one. We note that we obtained the bimodal 2-convex generating function 
in Section 2. We therefore obtain the generating function for 2-convex polygons 
by summing the results, multiplying each term by the cardinality of the class it 
counts. This gives the following generating function for 2-convex polygons: 
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where A^^^ = (1 - x)^^x^ - 3(1 - x)^x'^i5 - 2x + x'^)y + [1 - x^x^iA - 12x + lOSx^ - 
79x3 ^1^4 _ i-^^^b _^ 32,6)j^2 x)^x'^{40 - 124a; + 455a;2 - 533x^ + 2Mx^ - %%x^ - 

2\x^ - 3a;^ -h a;*)^^ -h (1 - xf{\ - 12x + 232x^ - 742x^ + 1696a;4 - 2297a;-^ + 1626a;6 - 
539x^ -I- 133a;^ -|- 3a;'' - 56x^^ + 3a;")?/4 + (1 - .t)^(-11 + 119.a; - 943.a;2 + 2U3x^ - 
lOlla;" + 4513x5 - 3054a;6 + 867a;^ - 58a;* + 22lx^ - 137x^" - 11a;" + x^'^)y'^ + (1 - 
a;)(55 - 542a; + 2765x^ - 6154a;3 + 8193a;4 - 7901a;5 + 5521a;6 - 2140.a;^ - 284a;8 + 430x^ + 
19x1° - Six" + 7x12)^6 ^ (■_ig5 ^ 1503a. _ 5996x2 ^ ii929a;3 _ i4004x^ -|- 11488x5- 
7661x^ + 4474,a;^ - 1456x* - 506x9 ^_ 5042;io _ leOx" + 18x^^)y'^ + (330 - 2502x -|- 
7381x2 - 10693X''' + 8925x'' - 4846x5 + 1856x6 - 1456x^ + 1364x* - 328x3 + 10Sx^° - 
22x")t/^ + (-462 + 2898x - GSSSx^ -|- 6348x3 - 3639x^ + 1204x5 -|- 714x<^ - 506x^ - 
328x8 - 32x3 + 10xi°)y3 + (462 - 2394x + 3925x2 - 2349x3 + 873x4 _ 537^5 „ ^^^6 _^ 
504x^ + 103x8 + lOx^)?/!" + (-330 + 1422x - 1797x2 + 434x3 + A7x* + 484x5 - lOOx^ - 
160x^ - 22x8)y" -|- (165 - 603x + 632x2 - 4x3 - 180x^ - 114x5 + 88x6 _^ i8x'^)yi-2 _ 
(1 - x)2(55 - 67x - 18x2 + 27x3 + 7x^)y'^^ - (1 - x)3(-ll + x2)y" _ (1 _ x)3yi5 
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and B^^"' = 4{l-xf^x^ -4:(l-x)'^'^x*{3Q-20x+7x'^)y+A{l-x)^^x^{4-Ux + 433x^- 
580x3 + 368a;''-128x5 + 21x6)y2-4(l-x)20a;2(100-360x + 4214a;2-8192a;3 + 7616a;^- 
4297a;5 + 1528x'^ - 350a;^ + 35x^)y^ + 4(1 - x)^^{l - 12x + YS/lx^ - 5226a;3 + 32426a:^ - 
76437x5 + 93156.x« - 70591x^ + 36061x*^ - 12917x9 + 3217xi" - 532x" + 35x^2)^4 _ 
(1 - x)i'^(104 - 1296x + 42936x2 - 198752x3 + 859020x'* - 2164230x5 + 3180634x^ - 
3007804x^ + 1964426x« - 927123x9 + 322834x1° - 83390x" + 15842x^2 _ igeix^^ + 
84xi^)?/5 + (1 - x)" (1300- 16868X + 2891 12x2- 1412308x3 + 5065708x4 _ i2664938x5 + 
20907814x^ - 23411754x^ + 18540432x8 - 10767204x'3 + 4716295x1° - 1577652x" + 
403402x^2 - 79358x13 + 12073x1^ - 1114x^5 + 'l%x^^)y'^ - (1 - x)i2(l0400 - 140832x+ 
1632176x2 - 8077604x3 _^ 26714752x4 - 64771716x5 + 113604188x6 - 143655868x^ + 
132836186x8 - 91595522x9 + 48102042x1° - 19641626xii + 6301753x12 - 1574826x13 + 
304364x" - 47486x15 + 6143x1^ - 400x1^" + 4xi8)y7 _ (i _ x)i" (-59800 + 846768x- 
7892920x2 + 38780872x3 - 125239164x4 + 297723354x5 - 537008734x'^ + 736108516x^ - 
767512896x8 + 611776002x9 - 375779397x1° ^ 180183183xii - 68757620x12 + 
21225217x13 - 5236463x14 + 983039x15 - 136742x"5 + 16579xi^ - 1952x1* + 
78x19)2/8 + (1 - x)8 (-263120 + 3902272x - 32692880x2 + 159495728x3 - 
518222904x4 + 1235851899x5 - 2273834564x° + 3286427610x^ - 3744105010x8 + 
3360169186x9 - 2372247082x1° + 1316981454xii - 578123374x12 + 204616292x13 - 
60356681x14 ^ I5025753xi5 - 2968864x1^ + 408498xi^ - 36191xi8 + 3653xi9 - 
450x2° + 7x21)2/9 + (1 - x)6(920920 - 14323848x + 115445704x2 - 565821908x3 + 
1881114592x4 - 4595055622x5 + 8689038821x*5 - 13096803662x^ + 15935786206x8 - 
15697620474x9 + 12485332431x1°- 7965917640x11 + 4040636749x12- 1619517767x13 + 
518745262x14 - 140030433x15 + 34601374x1° - 7726101x1^ + 1290999x18 - 
115181x19 + 1171x2° - 68x21 ^ 43a;22)yio + (i _ x)4(-2631200 + 42965472x - 
346781248x2 + 1732729172x3 - 5960460908x4 + 15133786878x5 - 29768776966x° + 
46871962702x^-60257832295x8+63871376160x9-55950160750xi°+40346497476xii- 
23700589523x12+11139364197x13-4087427426x14+1145559467x15-251575315x16 + 
52651169x1^ - 13478667x18 + 3310337x19 - 518486x2° + 32936x2i + 909x22 _^ 
2x23 + 3x24)2/11 - (1 - x)3 (-6249100 + 100966228x - 786008036x2 _,_ 3790621220x3 - 
12634194004x4+31120468774x5-59333283779x6+90481426644x^-112836337344x8+ 
116619494723x9 - 100526269572x1° + 72257250633xii - 42935196678x12 + 
20646926051x13 - 7717297655x14 + 2081066620x15 - 344551302x1° + 19708982x" - 
190245x18 + 2604586x19 - 1089925x2° + 167 9 90x21 - 53 25x22 - 8 1 9x23 + 5x24)2/12 - (1 - 
x)2 (12498200 - 200432672X + 1523814896x2 - 7173568104x3 + 23394202064x4 - 
56432397385x5 + 105258925336x° - 156756419659x^ + 190737577732x8 - 
192647570536x9 + 163201860685x1° - 116641434399xii + 70265808073x12 - 
35189781755x13 + 14129858643x14 - 4206300270x15 + 752961328x1° + 2762280x1''- 
41077210x18 + 8781488x19 + 172960x2° - 372435x21 + 59206x22 - 2116x23 - 
89x24 ^ 11x25)2/13 - (1 - x)(-21246940 + 339297288x - 2543202516x2 + 
11796257160x3-37956243060x4+90389633588x5-166278246916x°+243693863948x^- 
291099521233x8 + 288163147560x9 - 239402103854x1° + 168782468152xii - 
101938448323x12 + 52836781903x13 - 23089813175x14 + 8076150924x15 - 
2006509792x16+229306444x1^+52941397x18-29443771x19+5233574x20-137308x21- 
96084x22 + 13023x23 - 102x24 + Z7,x'^>)y^^ + (-30904640 + 493047872x - 
3671274848x2+16909765768x3-54076869168x4+128034624944x5-233957204524x6+ 
339860131708x^ - 401161475162x8 + 390960587591x9 - 318456383150x1° + 
219480309921x11 - 130108934416x12 + 67655799311x13 - 31165964099x14 + 
12455845878x15 - 4042728520x1° + 929084638x" - 93195098x18 - 24059794x19 + 
11705293x2° - 198 3 6 52x21 + 86 5 82x22 + 23 5 47x23 - 3 151x24 - 68x25 + 5x26)2/15 + 
(38630800 - 579105408X + 4018988224x2 - 17153382704x3 + 50579891852x4 - 
109880934764x5 + 183275392200x6 - 241692656260x^ + 257400864147x8 - 
224539648706x9 + 161936707748x1° - 97616424050xii + 50386570178x12 - 
23295420511x13 + 10082660716x14 - 4042728520x15 + 1381926138x16 - 352763968x1''+ 
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51556603xi« + 1631397x1'^ - 2559687x2" + 547928x21 - 54439x^2 - 61 Ix^^ + 432x^4 - 
31x25)yi6 + (-41602400 + 585761792 * x - 3791089760 * + 15002579968 * x^ - 
40793586584 * x^ + 81262920502 * x^ - 123537868000 * x^ + 147559428776x^ - 
141331365760x8 + 109736530568x'^ - 69193200719x1° + 35421383132x" - 
15013860403x12 + 5709460646x1^ - 2235816236x" + 929084638x1^ - 352763968x1^ + 
103409188x1^ - 19757856x1^ + 1570918x19 + 189358x2° - 71908x21 + 11883x22 - 
268x23 + 77x24)yi7 + (38630800 - 510758608x + 3084598320x2 - 11324232712x3 + 
28395037220x4 - 51826962892x5 + 71682098798x^ - 77353955608x^ + 66429125667x8 - 
45706907743x9 + 24906199156x1° - 10403200527xii + 3174037717x12 - 706359558x1^ ^ 
176365047x14 - 93195098x1^ + 51556603x1^ - 19757856xi^ + 4723092x1* - 605060x1'^ + 
49569x20-3395x21 - 1253x22 - 143x23)2/18 + (-30904640+383692992x- 2163792656x2+ 
7374341864x3 - 17053687520x4 + 28497776084x5 - 35790327136x<5 + 34800116028x^ - 
26729224736x8 + 16280988977x9 - 7648226004x1° + 2524992746xii - 406853569x12 - 
93698188x13 + 82385168x14 - 24059794x1^ + 1631397x1*^ + 1570918x" - 605060x18 + 
66554x19 - 4454x20 + 1819x2i + 202x22)y is + (21246940 - 247772008x + 1305971612x2 - 
4135556872x3 + 8823936024x4 - 13492690928x5 + 15356870874x6 - 13416475166x^ + 
9196907145x8 - 4978222336x9 + 2052527732xio - 556811827xii + 29183400x12 + 
58467226x13 - 34677345x14 + 11705293x15 - 2559687x1*^ + 189358xi^ + 49569x18 - 
44 54x19 - 1368x20 - 110x2i)y20 + (-12498200 + 136980272x - 675634696x2 + 
1990453776x3 - 3921712444x4 + 5486790816x5 - 5649513210x0 + 4421089360x^ - 
2695886222x8 + 1305174655x9 - 491084981x1° + 128534739xii - 11441768x12 - 
8063133x13 + 5370882x14 _ i983652xi5 + 547928xio - 71908x" - 3395xi8 + 1819xi9 - 
110x20)y2i + (6249100 - 64433028X + 297851620x2 - 817733112x3 + 1489860872x4 - 
1908408554x5 + 1775572130x6 - 1240442976x^ + 667928313x8 - 289317379x9 + 
102644601x10 - 29961766xii + 6383656x12 _ 977036x13 - 41224xi4 + 86582xi5 - 
54439x10 + 11883x" - 1253xi8 + 202xi9)y22 + (-2631200 + 25559072x - 110940720x2 + 
284446548x3 - 480234432x4 + 564112724x5 - 474248936x0 + 294904852x^ - 
138225516x8 + 52522411x9 - 17224508xio + 5578263xii - 1609051x12 + 492963xi3 - 
109107xi4+23547xi5-611xiO-268x"-143xi8)2/23_,_(920920-8444128x+34492144x2- 
82799336x3 + 129893340x4 - 140307634x5 + 106798702x0 - 59095890x^ + 23639299x8 - 
7486656x9 + 2001655xi" - 644781xii + 181503xi2 - 63349xi3 + 13125xi4 - 3151xi5 + 
432x10 + 77x")y24 + (-263120 + 2283072x - 8798288x2 + 19829008x3 - 28995448x4 + 
28906719x5-20012600x0+9913898x^-3347134x8+826000x9- 124087xi°+29300xii- 
2853x12 + 1927x13 - 135xi4 - 68xi5 - 31xi0)y25 + (598OO - 492568x + 1795744x2 - 
3809804x3 + 5207920x4 - 4809154x5 + 3046581x0 - 1369614x^ + 399732x8 - 78407x9 + 
2224x10 + 919x11 - 834xi2 + lllxi3 + 33xi4 + 5x15)2/26 + (_ 10400 + 81632x - 282336x2+ 
565028x3 - 723116x4 + 619222x5 - 359946xO + 148482x^ - 39609x8 _^ 74493,9 _ 326xi°- 
10x11 + 5x12 _ 112,13)^27 _^ (1300 _ 9768x + 32140x2 - 60712x3 ^ 72592^4 - 57298x5 + 
30217x0 - 11207x^ + 2732x8 - 506x9 ^ 433,10 ^ 3a;ii)y28 _^ (-_io4 + 752^; _ 2360x2 + 
4200x3 - 4648x4 + 3305x5 _ i506xO + 448x^ - 78x8 ^ 7a;9)y29 ^ _ xfyio^ 



A Enumerating staircase and unimodal polygons with 
m vertical indents on the left side 

Proposition A.l. The generating function for m- staircase (resp. m-unimodal) 
polygons with left-side vertical indents at different heights is 

(47) 

m! oy dy y 

where I = (SP/y)^ and Q = SP/y (resp. UP/y). 



51 



Figure 26: The construction of 3-staircase polygons. Three vertical steps are 
selected as the heights of the indents. All indents are placed at the first choice. 

Proof. The form of 3-staircase polygons with only left-side vertical indents can 
be seen in Figure 26. This can be generalised to m-unimodal polygons. Staircase 
boxed regions are constructed as above, giving an m — 1 x m grid. By again 
factorising at the points of intersection, one can construct equivalence classes in 
terms of the boxes intersected by the bottom perimeter. 

For example, the generating function for all classes whose bottom perimeter 
intersects the n*'' box in the first row and does not intersect any box in the 
(n — l)**" column, but rather follows the n**" column down, out of the grid, can 
be written as 

\5y^-^) \Sy y )■ 

Hence, all classes of m-unimodal polygons that are defined by the intersec- 
tions of boxes have a generating function that is determined by the relative 
position of the first box intersected in each column. For example, the path il- 
lustrated in Figure 26 intersects the 3"''' box in the first row, and the 2""* box 
in the second row. This class could be uniquely described by the pair (1,2) of 
the relative positions of these boxes. In general, one needs an (m — l)-tuple to 
classify classes as such. Noting that symmetric classifications also have the same 
generating function, one can show that the cardinalities of the sets of classes 
with the same generating function match that of the required set of classes of 
m-unimodal polygons. □ 

Remark. The form of the generating function of m-unimodal polygons with only 
vertical indents on the left side is as follows: 

x{A + VAB) 

ym+l^m+1/2 ' 

where A (resp. B) is a polynomial of order 4m — 1 (resp. 4m — 2) in both x and 

y- 
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